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EXECUTIVE  SUMMARY 


Improving  the  current  air  traffic  system  with  new  technologies  is  necessary  to  reduce  or  manage 
current  Air  Traffic  Control  Specialists  (ATCSs)  workloads,  while  accommodating  continued 
growth  in  air  traffic.  As  the  National  Airspace  System  (NAS)  continues  to  evolve,  ATCSs’  roles 
and  responsibilities  will  change  to  adapt  to  the  implementation  of  new,  automated  tools  and 
procedures.  The  introduction  of  automated  decision  support  tools  (DSTs)  will  allow  the  ATCS 
to  move  from  tactical  control  of  traffic  to  more  strategic  control  of  traffic.  DSTs  identify 
potential  conflicts  and,  in  the  future,  will  provide  solutions  to  potential  problems.  Expected 
benefits  from  these  new  automated  tools  include  increased  safety,  efficiency,  and  throughput. 

Although  research  into  the  effects  of  automation  on  job  performance  exists,  we  know  little  about 
the  effects  of  DSTs  on  performance  of  ATCSs.  The  introduction  of  DSTs  into  the  Air  Traffic 
Control  (ATC)  environment  will  change  the  way  ATCSs  work.  We  have  only  limited 
understanding  of  how  much  DSTs  will  affect  ATCS  strategies  and  procedures  because  of  1) 
limited  applied  research,  2)  limited  acquisition-funded  studies  that  address  specific  risks,  and  3) 
operational  field  trials  without  established  baselines.  Although  we  expect  DSTs  to  improve 
safety  and  efficiency,  it  is  important  to  investigate  the  effects  of  these  anticipated  changes  on  the 
ATCSs’  behaviors.  Through  careful,  controlled  experimentation,  researchers  gain  information 
about  the  possible  consequences  of  changes  in  the  ATC  environment.  The  results  obtained  from 
experimentation  can  then  guide  development  of  the  most  appropriate  and  effective  strategies  to 
offset  any  detrimental  consequences  from  these  changes. 

The  current  study  examines  the  effects  of  automation  and  traffic  volume  on  ATCSs  in  an  en 
route  ATC  environment.  To  mimic  the  predicted  increases  in  air  traffic,  we  manipulated  task 
load  based  on  the  volume  of  air  traffic.  ATCSs  controlled  both  low  and  high  traffic  task  loads. 
We  also  manipulated  the  level  of  automation  ATCSs  used  in  controlling  traffic.  This  included 
three  levels  ranging  from  no  automation  to  limited  automation  to  full  automation.  Further, 
ATCSs  rotated  between  the  Radar  (R-side)  and  Data  (D-side)  positions. 

We  anticipated  changes  in  ATCSs’  behaviors  with  the  manipulation  of  automation,  task  load, 
and  ATCS  position.  We  examined  ATCSs’  behavior  and  cognitive  processing  through  objective 
and  subjective  performance  and  behavioral  measures.  These  measures  examined  ATCSs’ 
situation  awareness  (SA),  workload,  visual  scanning,  communications,  and  trust. 

The  execution  of  this  experiment  was  an  achievement  and  required  extensive  cross-disciplinary 
teaming  and  new  operational  procedures.  To  pull  together  the  different  disciplines  and 
organizations,  we  created  a  crosscutting  team  that  involved  test  and  evaluation,  design  and 
engineering,  simulation,  and  human  factors  groups  that  spoke  a  common  language  and  worked 
together  to  make  things  happen.  For  this  experiment,  we  developed  Generic  Center  Airspace  on 
the  Host  Computer  System  and  the  DST.  Although  we  had  used  our  human  factors  instruments 
in  other  studies,  we  transferred  them  to  operational  hardware  and  software.  Finally,  to  better 
assess  SA,  we  developed  a  new  SA  assessment  instrument  called  the  SA  Verification  and 
ANalysis  Tool  (SAVANT)  that  used  data  from  the  operational  systems  to  probe  ATCS  SA. 
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Sixteen  ATCSs  from  Air  Route  Traffic  Control  Centers  within  the  United  States  voluntarily 
participated  in  the  experiment  conducted  at  the  William  J.  Hughes  Technical  Center  in  Atlantic 
City,  NJ.  ATCSs  performed  en  route  simulations  under  low  and  high  task  load;  no,  limited,  and 
full  automation;  and  in  the  R-  and  D-side  positions.  ATCSs  controlled  traffic  in  a  generic 
airspace  to  make  our  findings  easy  to  generalize  and  to  increase  the  size  of  our  potential 
participant  pool.  ATCSs  received  training  on  the  generic  airspace,  the  use  of  the  DST  (including 
hands-on  training),  and  all  equipment  used  in  the  simulation  prior  to  experimental  runs.  We 
assessed  SA  using  SAVANT,  SA  Global  Assessment  Technique,  self-report  measures,  and  over- 
the-shoulder  (OTS)  ratings  by  an  ATC  Subject  Matter  Expert.  We  obtained  workload  ratings 
from  the  Air  Traffic  Workload  Input  Technique  (ATWIT),  National  Aeronautics  and  Space 
Administration’s  (NASA)  Task  load  Index  (TLX),  and  self-report  measures.  An  eye  tracking 
system  collected  visual  scanning  data.  We  examined  ATCSs’  landline  and  air-to-ground 
communications  via  push-to-talk  software  and  intrateam  communication  via  coding  of  digitized 
audio  and  video  using  the  Federal  Aviation  Administration’s  Controller-to-Controller 
Communications/Coordination  Taxonomy.  ATCSs  expressed  their  level  of  trust  in  the  DST  on 
self-report  questionnaires.  Post-Scenario  Questionnaires  provided  self-report  data  from  the 
ATCSs,  and  OTS  ratings  provided  subjective  performance  data. 

ATCS  SA  was  lower  with  limited  automation  than  it  was  with  or  without  full  automation.  The 
subjective  ratings  on  SA  for  potential  violations  and  our  objective  SA  measures  showed  that 
ATCSs  displayed  lower  SA  with  full  automation  and  under  high  traffic  task  load  conditions. 

This  finding  shows  that  the  benefit  that  automation  provides  under  low  traffic  task  load 
conditions  no  longer  offsets  the  loss  of  D-side  assistance  on  radar  tasks.  The  introduction  of 
automation  may  seem  to  have  a  benefit  under  low  traffic  task  load  conditions,  but  it  may  disrupt 
that  balance  between  R  and  D  positions  under  high  traffic  task  load.  We  see  this  in  the 
coordination  between  sectors.  Under  full  automation  and  low  task  load,  ATCSs  spent  a 
considerably  longer  amount  of  time  communicating  with  the  next  sector  than  under  full 
automation  and  high  task  load.  Two  explanations  may  account  for  this  finding.  First,  when  task 
load  was  low,  ATCSs  had  enough  time  to  conceptualize  detailed  plans  to  solve  potential 
problems  and  discussed  them  in  detail  with  the  adjacent,  controlling  sector.  In  contrast,  high  task 
loads  led  the  R-side  ATCS  to  need  more  assistance  from  the  D-side  and  pulled  the  D-side  away 
from  strategic  planning  using  the  DST.  In  the  analyses  of  most  of  our  data  sets,  we  observed 
with  automation,  the  D-side  being  pulled  away  from  his  or  her  traditional  function  of  immediate 
assistance  to  the  R-side.  The  ATC  culture  uses  the  D-side  for  immediate  tactical  assistance  when 
traffic  levels  are  high.  Therefore,  without  changing  the  roles  and  responsibilities  to  include  a 
strategic  use  of  a  DST,  the  D-side  will  have  to  choose  between  tasks  and,  under  high  traffic 
levels,  the  DST  will  likely  go  unused. 

ATCSs  indicated  that  the  full  automation  conditions  were  less  realistic  and  more  difficult  than 
the  conditions  without  automation.  Scenarios  without  automation  mimic  the  current  ATC  field 
environment  that  utilizes  flight  strips,  but  no  DSTs.  The  perceived  difficulty  of  the  DST  may  be 
because  the  ATCSs  were  novice  DST  users,  and  they  needed  to  use  conscious  effort  to  use  the 
DST.  Other  over  learned  behaviors  (e.g.,  use  of  flight  strips)  are  automated  and  effortless. 
Besides  increasing  the  perceived  difficulty  ratings  for  the  D-side  ATCS,  this  also  may  have  led 
to  an  indirect  increase  in  the  perceived  difficulty  for  the  R-side  ATCS. 
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Task  load  had  an  impact  on  most  of  the  dependent  measures.  ATCSs  rated  the  low  task  load 
scenarios  more  representative  and  closer  to  their  normal  traffic  levels.  Although  ATCSs  did  not 
view  the  high  task  load  scenarios  as  representative  of  what  they  currently  experience  in  the  field, 
the  results  for  the  high  task  load  scenarios  may  give  us  insight  into  what  to  expect  when  traffic 
levels  increase  in  the  field.  Ground-to-air  communications  increased  in  number  and  decreased  in 
duration  under  high  task  loads,  whereas  coordination  using  the  landline  increased.  Within  the 
team,  ATCSs  increased  the  number  of  comments  regarding  specific  aircraft  and  altitude  changes 
as  task  load  increased.  Trust  in  the  DST  fluctuated  with  task  load  levels.  Although  the 
algorithms  in  the  DST  did  not  change  between  conditions,  ATCSs  indicated  that  they  trusted  the 
DST  the  least  under  high  task  load  conditions.  It  is  under  high  task  load  conditions  that  the 
DST’s  presence  is  most  needed;  however,  ATCSs  are  the  least  likely  to  trust  and  use  it. 

Workload  ratings,  as  measured  by  the  ATWIT,  NASA  TLX,  and  self-report  ratings,  indicated 
that  as  task  load  increased,  workload  increased.  Interestingly,  over  time,  workload  ratings 
decreased  under  low  task  load,  but  they  remained  relatively  constant  under  high  task  load.  These 
results  and  our  SA  findings  suggest  that  when  task  load  was  low,  ATCSs  were  able  to  catch  up 
and  preplan  control  actions,  decreasing  workload  over  time.  High  task  load  conditions  did  not 
allow  for  this. 

ATCSs  and  the  OTS  raters  felt  that  an  increase  in  task  load  resulted  in  decreased  SA  and 
performance  in  general.  Under  high  task  load  conditions,  the  increased  number  of  aircraft  made 
it  difficult  for  ATCSs  to  maintain  the  “picture,”  as  reflected  in  each  SA  measure.  Future  sector- 
based  SA  was  better,  awareness  for  aircraft  callsign  letter  and  aircraft  separation  was  higher, 
self-report  SA  was  higher,  and  OTS  (subjective  performance)  ratings  were  higher  when  task  load 
was  low.  In  contrast,  SA  was  higher  for  awareness  of  aircraft  clearances  received  when  task  load 
was  high.  Interestingly,  our  simulator  data  showed  that  ATCSs  did  attempt  to  setup  the  high  task 
load  scenarios  to  become  more  structured,  potentially  reducing  the  loss  of  SA. 

We  used  our  ATCS  participants  as  their  own  controls,  that  is,  they  worked  under  the  same 
conditions  both  as  a  R-side  and  a  D-side.  When  working  on  the  R-side,  ATCSs’  SA  was  better 
than  when  on  the  D-side  was.  Of  course,  the  awareness  for  the  tactical  situation  of  a  D-side  can 
be  less  than  that  of  an  R-side  because  the  D-side  ATCS  is  less  involved  in  tactical  control  of  the 
sector. 

Although  the  D-side  ATCS  is  an  integral  part  of  the  sector  team,  our  participants  indicated  that 
they  felt  they  controlled  traffic  less  well  when  working  as  the  D-side  ATCS.  These  results 
reflect  that  the  D-side  does  not  directly  control  traffic,  but,  because  our  participants  were  all 
certified  R-side  ATCSs,  they  felt  that  they  could  have  been  of  more  assistance  if  they  would 
have  had  R-side  responsibilities.  When  working  the  D-side,  our  ATCS  participants  also 
indicated  that  high  traffic  task  loads  resulted  in  less  of  an  increase  in  workload  than  when 
working  the  R-side.  This  may  reflect  spare  capacity  within  the  sector  team  on  the  part  of  the 
D-side. 
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1.  INTRODUCTION 


The  introduction  of  new  technologies  into  the  National  Airspace  System  (NAS)  will  significantly 
alter  the  role  of  the  Air  Traffic  Control  Specialist  (ATCS).  As  the  Federal  Aviation 
Administration  (FAA)  moves  forward  with  NAS  modernization  and  the  implementation  of  Free 
Flight  concepts,  the  ATCS  will  move  from  a  controlling  role  to  a  collaborating  role  (Office  of 
Air  Traffic  System  Development,  1997).  With  increases  of  up  to  60%  expected  in  air  traffic  over 
the  next  decade,  automation  will  play  a  significant  role  in  supporting  the  NAS  and  the  ATCS 
(Office  of  Air  Traffic  System  Development).  The  ATCS  will  be  able  to  approve  more  user 
requested  routes  with  the  assistance  of  automated  decision  support  tools  (DSTs)  (Federal 
Aviation  Administration,  1999).  As  outlined  by  Kirk,  Heagy,  and  Yablonski  (2000),  the 
anticipated  benefits  from  the  use  of  DSTs  will  encompass  increases  in  safety,  efficiency,  and 
ATCS  productivity. 

We  performed  the  current  study  to  gain  a  better  understanding  of  the  impact  of  DSTs  on  ATCSs’ 
cognitive  and  behavioral  functioning.  This  study  examined  the  effects  of  automation  levels  and 
increases  in  air  traffic  volume  on  ATCSs’  situational  awareness  (SA),  workload,  visual  scanning, 
communications,  and  the  safety  and  efficiency  with  which  they  controlled  traffic. 

This  study  took  place  at  the  FAA  Research  Development  and  Human  Factors  Laboratory 
(RDHFL)  and  the  Integration  and  Interoperability  Facility  (I2F)  in  Atlantic  City  International 
Airport,  NJ.  The  experiment  involved  engineers  and  researchers  from  I2F  Support  Services  of 
the  En  Route  Branch  (ACT-233),  NAS  Simulation  Branch  of  the  NAS  System  Engineering  and 
Analysis  Division  (ACT-510),  NAS  Human  Factors  Branch  of  the  NAS  System  Engineering 
Analysis  Division  (ACT-530),  Civil  Aeromedical  Institute  (CAMI),  and  their  contractors. 

1.1  Background 

A  primary  goal  of  Air  Traffic  Control  (ATC)  is  safety.  ATC-related  aircraft  accidents  are  rare; 
however,  at  least  one  study  has  attributed  over  90%  of  all  errors  to  either  ATCSs  or  supervisors 
(Kinney,  Spahn,  &  Amato,  1977).  Operating  under  high  workloads,  tight  regulations,  and 
challenging  conditions,  the  ATCS  job  is  both  demanding  and  stressful  (Finkelman  &  Kirschner, 
1980).  These  conditions  are  likely  to  induce  errors. 

Improving  the  current  ATC  system  with  new  technologies  is  necessary  because  of  the  projected 
increase  in  air  traffic.  These  new  technologies  include  the  introduction  of  additional  automated 
tools  for  all  phases  of  flight.  The  FAA  Office  of  Air  Traffic  System  Development  (1997)  has 
laid  out  a  plan  that  includes  these  efforts.  Expected  improvements  from  the  new  automation 
include  increased  efficiency  and  throughput.  Table  1  lists  these  automation  efforts  in  ATC 
(Gosling,  1992;  Zweben,  1992). 
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Table  1 .  Active  or  Planned  Automation  Efforts 


• 

Conflict  detection  and  prediction 

• 

Weather  forecasting 

• 

Conflict  resolution 

• 

Flight  progress  monitoring 

• 

Automated  flight  planning  &  scheduling 

• 

Situation  classification 

• 

Approach  guidance 

• 

Data  fusion 

• 

Departure  guidance 

• 

System  function  allocation 

• 

• 

Surface  control  management 

ATCS  training 

• 

Modeling  of  the  ATCS 

The  aviation  industry  has  introduced  automation  into  the  working  environment  with  the  goal  of 
increasing  safety  and  efficiency  in  addition  to  supporting  the  continued  growth  of  air  traffic. 
Although  extensive  research  on  the  general  effects  of  automation  on  behavior  exists,  we  know 
little  about  the  effect  of  the  introduction  of  automated  DSTs  on  ATCSs.  In  their  current 
capacity,  ATCSs  have  developed  strategies,  procedures,  and  conscious,  as  well  as  automated 
behaviors  to  safely  and  efficiently  direct  traffic.  These  strategies  include  what  has  worked  in  the 
past  for  the  ATCS,  such  as  similarity  of  traffic  flow  through  airspace.  Facilities  establish 
procedures  that  make  traffic  most  efficient  (e.g.,  silent  coordination  between  Terminal  Radar 
Approach  Control  (TRACON)  and  low  altitude  en  route  sector.  This  may  include  handoffs  of 
aircraft  that  are  suppose  to  climb  to  a  certain  altitude  at  a  certain  speed  -  ATCSs  then  do  not 
need  to  use  the  landline  (LL)  to  coordinate  this).  In  new  situations,  ATCSs  need  to  revert  to 
consciously  analyzing  the  situation  to  arrive  at  a  solution  that  may  work.  In  contrast, 
recognition-primed  decision  making  (Klein,  1999)  is  an  automated  ATCS  behavior  where 
ATCSs  recognize  the  solution  to  a  problem  more  than  the  problem  itself.  The  introduction  of 
new  tools  will  change  the  way  they  operate.  Questions  arise  about  the  effect  these  new  tools  will 
have  on  the  ATCSs.  We  expect  that  the  introduction  of  automated  DSTs  in  the  en  route 
environment  will  alter  ATCSs’  visual  scanning,  workload,  and  SA.  These  changes  may  in  turn 
contribute  to  the  overall  safety  and  efficiency  of  the  NAS. 

1.1.1  Automation  Issues 


As  the  ATC  system  evolves  to  meet  future  increased  traffic  levels,  automation  will  become 
increasingly  more  important.  The  automated  tools  presently  in  use  include  RADAR,  Host, 
Display  System  Replacement  (DSR),  User  Request  Evaluation  Tool  (URET),  Voice  Switching 
and  Communication  System  (VSCS),  Conflict  Alert  (CA),  and  Minimum  Safe  Altitude  Warning 
(MSAW).  Ongoing  projects  that  will  result  in  new  automated  tools  for  future  use  include  Direct 
To  (D2),  Traffic  Management  Advisor  (TMA),  En  Route  Descent  Advisor  (EDA),  and  Problem, 
Analysis,  Resolution,  and  Ranking  (PARR). 

The  literature  widely  discusses  the  technical  and  practical  considerations  of  automation  (Ignizio, 
1991;  Waterman,  1986)  but  does  not  address  how  it  produces  new  problems  for  human 
performance  (Endsley,  1993).  As  we  continue  to  rely  on  the  support  of  automation,  it  is  essential 
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to  better  understand  its  impact  on  human  performance.  As  errors  involving  automated  systems 
tend  to  be  more  cataclysmic  and  costly  (Wickens,  1992),  the  human  interface  has  become  more 
important  than  ever. 

1 . 1 . 1 . 1  Effects  of  Increased  System  Monitoring  Task  Load  on  ATCSs 

Automation  leaves  the  human  with  fewer  functions  but  with  a  more  complex  system  to  monitor  - 
a  role  in  which  people  do  not  excel  (Wickens,  1992).  Out-of-the-loop  performance  problems  are 
a  major  class  of  errors  associated  with  automation.  Operators  who  are  out-of-the-loop  are  slower 
and  less  accurate  at  failure  detection  because  they  are  passive  decision-makers  (Wickens  & 
Kessel,  1979;  Wickens  &  Kessel,  1980;  Young,  1969).  Human  monitors  have  problems 
detecting  system  errors  and  performing  tasks  manually  in  the  event  of  automation  failure 
(Billings,  1988;  Wickens,  1992;  Wiener  &  Curry,  1980).  In  a  review  of  automation  problems, 
Billings  traced  six  major  aircraft  accidents  directly  to  failures  to  monitor  the  status  of  an 
automated  system.  In  addition  to  delays  in  detecting  that  a  problem  has  occurred,  human 
operators  may  require  a  significant  period  of  time  to  develop  a  sufficient  understanding  of  the 
state  of  the  system  to  act  appropriately.  This  delay  may  prohibit  them  from  carrying  out  their 
tasks  or  it  may  diminish  their  effectiveness  (e.g.,  National  Transportation  Safety  Board,  1990). 
We  attribute  these  problems  to  lower  levels  of  operator  SA  that  can  occur  with  automation 
approaches  that  place  people  in  the  role  of  passive  monitor.  In  addition,  a  loss  of  skills  needed  to 
take  over  manually  may  result  (Wiener  &  Curry). 

1 . 1 . 1 .2  Effects  of  System  Complexity  on  ATCSs 

Automation  tends  to  increase  system  complexity.  The  increased  probability  of  system  failure 
related  to  the  increased  number  of  systems  adds  to  the  complexity  of  the  user's  job  (Wickens, 
1992).  In  addition,  complex  systems  may  increase  workload,  make  the  system  more  difficult  to 
learn  (Scerbo,  1996),  and  negatively  affect  SA  (Endsley,  1999).  The  user  must  be  aware  of  what 
mode  is  active  and  the  procedures  associated  with  all  modes  of  the  automated  system  (Scerbo). 

It  may  not  be  clear  to  the  user  what  components  are  working,  and  the  user  may  create  his  or  her 
own  model  for  the  inner  workings  of  the  system.  This  makes  the  cause  and  effect  relationship  of 
the  system  less  clear.  The  user  may  become  “surprised”  or  “suspicious”  of  the  automation  when 
the  user  cannot  account  for  an  action  taken  by  the  automation  that  appears,  to  the  user,  to  be 
illegitimate  (Wickens  &  Hollands,  2000).  This  can  lead  to  negative  consequences  if  the  user 
takes  inappropriate  actions  to  correct  the  perceived  problem  (Wickens  &  Hollands). 

1 . 1 . 1 .3  Effects  on  ATCS  Decision  Making 

It  is  unclear  whether  automated  systems  will  improve  the  quality  of  ATCS  decisions.  Increasing 
evidence  suggests  that  ATCS  performance  does  not  benefit  from  the  addition  of  traditional 
decision  aids  (Endsley  &  Kaber,  1996;  Endsley  &  Kiris,  1995;  Kibbe,  1988;  Selcon,  1990).  For 
example,  the  ATCS  must  make  a  comparison  between  his  or  her  own  way  of  doing  the  task  and 
the  way  the  system  does  the  task  and  decide  whether  to  accept  the  recommendations  of  the 
system.  This  can  all  take  additional  time,  particularly  if  information  is  ambiguous.  If  the  system 
advice  is  ambiguous  or  incorrect,  the  system  is  generally  more  likely  to  reduce  human  decision 
quality  and  speed. 
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Decision  support  advice  represents  a  new  source  of  information,  adding  to  the  decision-making 
problem  as  much  as  assisting  it  (Endsley  &  Selcon,  1997;  Pritchett,  1997).  With  the  presence  of 
a  decision  aid,  the  user  may  experience  problems  stemming  from  limited  knowledge  or 
misperceptions  of  how  the  automated  system  works  (Mosier  &  Skitka,  1996).  Further,  as  noted 
by  Mosier  and  Skitka,  automated  decision  aids  provide  another  type  of  decision  heuristic  from 
which  a  decision  maker  can  act.  In  the  context  of  ATC,  this  implies  that  an  ATCS  using  a  DST 
may  accept  the  DST  information  without  more  in-depth  analysis  of  the  situation,  may  discount 
any  conflicting  information,  or  ignore  other  pertinent  information.  All  of  these  have  negative 
effects  on  ATCS  SA  and  performance. 

1 . 1 . 1 .4  Effects  on  ATCS  Workload  and  Performance 

Wickens  and  Hollands  (2000)  noted  that  automation,  when  properly  implemented,  reduces 
workload  involving  muscular  exertion,  decision  choice,  and  information  acquisition  and  analysis. 
As  applied  to  ATC,  we  would  expect  a  DST  to  reduce  ATCS  workload  initially  through 
information  acquisition  and  analysis  and  then,  with  future  versions,  decision  choice.  The 
purpose  of  one  DST  is  to  predict  and  identify  future  traffic  conflicts  with  a  20-minute  look-ahead 
time.  The  ATCS  then  formulates  a  decision  to  correct  the  conflict  and  can  check  it  against  the 
DST.  Future  DSTs  will  be  more  active  and  will  compute  a  corrective  action  that  the  ATCS  then 
implements. 

Although  automation  has  focused  on  reducing  workload,  Hart  and  Sheridan  (1984)  noted  that 
automation  often  replaces  workload  involving  physical  activity  with  workload  involving 
cognitive  and  perceptual  activity.  In  ATC,  workload  will  likely  shift  from  one  task  to  another, 
although  the  tasks  will  continue  to  be  of  a  mostly  cognitive  and  perceptual  nature.  Wiener’s 
(1985)  studies  in  commercial  aviation  found  a  significant  number  of  pilots  reporting  that 
automation  did  not  reduce  their  workload  but  increased  it  during  critical  portions  of  the  flight. 

Operator  monitoring  of  the  system,  a  component  of  automation,  may  induce  high  workload. 

When  people  use  sustained  attention  to  monitor,  it  induces  considerable  fatigue  (Galinsky,  Rosa, 
Warm,  &  Dember,  1993)  and  high-perceived  workload  (Becker,  Warm,  &  Dember,  1991; 
Dittmar,  Warm,  Dember,  &  Ricks,  1993;  Scerbo,  Greenwald,  &  Sawin,  1993).  Even  when  task 
load  is  not  high,  the  requirement  to  vigilantly  monitor  automated  systems  imposes  its  own 
workload.  In  studies  involving  cockpit  automation,  pilots  complained  that  automation  required 
constant  scanning,  adding  to  workload  (Wiener,  1985).  For  the  ATC  environment,  these 
findings  indicate  that  the  addition  of  automation  for  many  tasks  may  only  shift  the  ATCS 
workload  from  monitoring  traffic  to  monitoring  both  the  automation  and  the  traffic.  This  may 
result  in  anything  from  equal  or  higher  workload  under  normal  conditions  to  increases  in 
workload  under  conditions  where  the  automation  has  problems  or  traffic  is  unusual. 

Harris,  Goemert,  Hancock,  and  Arthur  (1994)  found  that  when  operators  had  to  initiate  task 
automation  in  response  to  an  unanticipated  increase  in  workload,  they  showed  an  increase  in 
performance  error  on  other  manual  tasks.  This  confirmed  work  by  Parasuraman,  Bhari,  Deaton, 
Morrison,  and  Barnes  (1992)  indicating  that  operator  initiation  of  automation  was  likely  to 
increase  demands  when  they  were  already  high.  This  may  explain  why  a  paradox  of  automation 
exists.  Operators  indicate  that  under  high  workload  they  frequently  turn  the  automation  off 
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(Wiener,  1988).  For  the  ATC  environment,  these  findings  indicate  that  under  high  traffic  task 
loads,  ATCSs  may  either  disregard  the  automation  or  focus  on  it  to  the  extent  that  their  other 
tasks  may  suffer. 

1 . 1 . 1 .5  Effects  on  ATCS  Situation  Awareness 

The  effect  of  automation  on  ATCSs’  SA  is  mixed.  According  to  Wiener  (1992)  and  Billings 
(1991),  when  appropriate  integrated  information  is  presented  to  the  user,  improvements  in  user 
S  A  may  occur.  However,  negative  effects  of  automation  on  user  SA  have  been  documented. 
Under  automation,  vigilance  decrements,  complacency,  or  a  lack  of  trust  in  automation  cause 
problems  with  SA.  People  may  neglect  monitoring  tasks,  attempt  to  monitor  but  do  so  poorly,  or 
be  aware  of  indicated  problems  but  neglect  them  due  to  high  false  alarm  rates.  Passive 
processing  of  information  under  automation  can  make  the  dynamic  update  and  integration  of 
system  information  more  difficult,  decreasing  SA.  Changes  in  or  a  complete  loss  of  feedback 
frequently  occur  either  intentionally  or  inadvertently  with  many  automated  systems  (Endsley  & 
Kiris,  1995). 

1 . 1 . 1 .6  Effects  on  ATCS  Trust  in  the  Automation 

The  decision  to  rely  on  automation  can  be  one  of  the  most  crucial  decisions  the  operators  of  a 
complex  system  can  make.  Riley  (1994)  investigated  factors  that  influenced  when  people  would 
choose  to  initiate  automation.  In  these  studies,  the  choice  to  use  automation  did  not  relate  to 
workload  levels  but  rather  to  factors  such  as  reliability,  trust,  and  risk  (Muir,  1988;  Riley,  1994). 
An  operator  who  does  not  trust  a  system  may  be  more  likely  to  commit  errors  (Danaher,  1980; 
Lee  &  Moray,  1992;  Wiener  &  Curry,  1980).  He  or  she  may  not  use  the  system,  even  when  it 
might  be  beneficial,  and  experience  excessive  workload  because  of  this.  This  can  lead  to 
compromised  overall  system  performance  thereby  undermining  the  reason  for  the  automation. 

On  the  other  hand,  too  much  trust  in  an  automated  system  can  lead  to  errors  (Lee  &  Moray, 

1992;  Stokes  &  Kite,  1994).  The  user 

•  may  become  complacent  or  overconfident  in  the  automated  decisions  reached, 

•  may  not  properly  monitor  system  state, 

•  may  allow  the  system  to  perform  functions  best  left  to  the  user  or  to  another  aid,  and 

•  may  operate  the  system  in  ways  for  which  it  was  not  designed  or  when  faulty. 

1,1.2  Proposed  Solutions  for  Automation  Issues 

The  issues  related  to  automation  have  led  to  proposed  solutions  such  as  “human-centered 
automation,”  support  of  SA,  optimal  level  of  automation,  flexible  function  allocation,  and 
relevant  feedback  provision. 

Automation  problems  have  led  to  human-centered  automation  to  overcome  the  limitations  of 
technology-centered  approaches  (Billings,  1988).  With  a  human-centered  approach,  human  roles 
have  a  higher  priority  (Hopkin,  1998)  and  give  the  user  the  greatest  amount  of  satisfaction 
(Wickens  &  Hollands,  2000).  In  this  context,  automation  is  designed  to  assist  the  user  in  his  or 


5 


her  tasks  (Hopkin).  In  ATC,  automation  has  often  lacked  a  human-centered  approach,  resulting 
in  rejection  of  the  automation  by  the  user  (e.g.,  Standard  Terminal  Automation  Replacement 
System  Human  Factors  Team,  1997).  ATCS-centered  automation  is  necessary  to  ensure  an  ATC 
system  that  provides  reliability,  performance,  safety,  and  acceptance.  These  efforts  must 
augment  the  ATCS  in  handling  the  predicted  increase  in  air  traffic. 

Wiener  and  Curry  (1980)  and  Billings  (1991)  discussed  the  implementation  of  automation  at 
various  levels.  Automation  has  an  impact  on  the  involvement  of  the  ATCS  in  the  decision 
process  and  thus,  on  performance  in  detecting  system  breakdowns  and  assuming  control. 
Automation  has  typically  resulted  in  decreased  SA  by  removing  the  operator  from  involvement 
in  system  operation.  An  alternate  approach  to  automation  keeps  the  ATCS  involved  by 
determining  a  level  of  automation  that  minimizes  negative  impacts  on  SA  (Endsley,  1987). 

Endsley  and  Kiris  (1995)  found  that  by  implementing  functions  at  a  lower  level  of  automation, 
the  operator  remained  involved  in  the  active  decision-making  loop,  SA  remained  at  a  higher 
level,  and  people  were  more  able  to  perform  the  task  manually  when  needed.  To  further  explore 
the  benefit  and  costs  of  intermediate  levels  of  automation  on  overall  human-machine 
performance,  Endsley  and  Kaber  (1999)  developed  a  more  detailed  level  of  automation 
taxonomy  (Table  2).  They  indicated  that  level  of  automation  significantly  affected  both  task 
performance  and  the  out-of-the-loop  performance  problem.  Automation  aided  the 
implementation  of  a  task  but  hindered  performance  when  involving  joint  human-automation 
option  generation.  Computer  aiding  at  the  action-selection  (decision  making)  part  of  a  task  did 
not  significantly  have  an  impact  on  performance  when  compared  to  purely  human  decision 
making.  Performance  at  the  high  end  of  the  level  of  automation  taxonomy  was  better  than 
manual  performance.  It  was,  however,  never  as  good  as  when  the  automation  only  assisted  in 
the  manual  implementation  aspects  of  the  task,  without  becoming  involved  in  the  higher-level 
cognitive  aspects. 

Human  ability  to  recover  from  and  perform  during  automation  failures  significantly  improved 
with  levels  of  automation  that  required  some  human  interaction  in  task  implementation. 
Following  an  automation  failure,  time-to-recover  and  manual  performance  was  worse  with  levels 
of  automation  that  allowed  advanced  queuing  of  targets.  Thus,  automation  strategies  that  allow 
operators  to  focus  significantly  in  advance  of  current  operations  may  contribute  to  out-of-the- 
loop  performance  decrements.  Workload  was  lower  and  SA  was  better  at  some  of  the  higher 
levels  of  automation. 

Interface  designs  may  need  to  support  adaptive  automation  (AA)  between  the  ATCS  and  the 
system  with  functions  being  passed  back  and  forth  as  circumstances  demand.  AA  allows  for 
sharing  of  control  between  the  user  and  the  automated  system  (Scerbo,  1996).  Past  systems  have 
not  supported  human  operators  in  alternately  assuming  or  delegating  tasks.  New  interfaces  need 
to  provide  the  SA  needed  to  support  control  transitions. 
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Table  2.  Level  of  Automation  Taxonomy  (from  Endsley  &  Kaber,  1999) 


ROLES 

LEVEL  OF  CONTROL 

MONITORING 

GENERATING 

SELECTING 

IMPLEMENTING 

the  System 

Alternative  Strategies 

a  Strategy 

a  Chosen  Strategy 

(1)  Manual  Control 

Human 

Human 

Human 

(2)  Action  Support 

Human/Computer 

Human 

Human 

Human/Computer 

(3)  Batch  Processing 

Human/Computer 

Human 

Human 

Computer 

(4)  Shared  control 

Human/Computer 

Human/Computer 

Human 

Human/Computer 

(5)  Decision  Support 

Human/Computer 

Human/Computer 

Human 

Computer 

(6)  Blended  Decision  Making 

Human/Computer 

Human/Computer 

Human/Computer 

Computer 

(7)  Rigid  System 

Human/Computer 

Computer 

buSniflHHH 

Computer 

(8)  Automated  Decision  Making 

Human/Computer 

Human/Computer 

isssssmmm 

Computer 

(9)  Supervisory  Control 

Human/Computer 

Computer 

Computer 

Computer 

(10)  Full  Automation 

Computer 

Computer 

Computer 

Computer 

The  system  design  needs  to  prevent  the  erosion  of  skills.  AA  may  aid  in  this  goal,  as 
demonstrated  by  Parasuraman  (1992).  AA  acts  to  help  maintain  operator  skills  and  performance 
levels  by  having  the  operator  take  over  manual  operations  at  regularly  scheduled  intervals. 


Endsley  (1996)  points  out  numerous  issues  concerning  AA.  First,  the  characteristics  of  tasks  that 
determine  their  optimal  level  of  control  and  suitability  for  AA  need  investigation.  Gluckman, 
Warm,  Dember,  and  Rosa,  (1993)  and  Carmody  and  Gluckman  (1993)  found  different  effects  on 
workload,  performance,  and  SA  for  AA  involving  a  static  (system  monitoring)  versus  a  dynamic 
(resource  management)  task.  Durso,  Gronlund,  and  Lewandowsky  (1993)  have  proposed  that  if 
automation  involves  parts  of  an  integrated  task,  more  performance  decrements  will  occur  than  if 
it  involves  the  whole  task.  Second,  examination  of  when  AA  should  change  its  level  of  support 
is  needed.  Parasuraman,  Molloy,  and  Singh  (1993)  examined  whether  manual  control 
implemented  at  a  pre-set  periodic  interval  differed  in  effect  from  manual  control  implemented 
based  on  poor  monitoring  performance.  They  found  no  differences  in  the  effect  on  subsequent 
human  monitoring  performance  under  automation.  The  insertion  of  a  period  of  manual  control 
was  equally  beneficial  in  both  cases.  Third,  a  major  question  lies  in  determining  how  to 
implement  AA  to  provide  the  potential  benefits  without  leading  to  loss  of  system  awareness. 
Many  systems  have  left  it  up  to  operators  to  invoke  automation  at  their  discretion.  Leaving  the 
system  with  the  ability  to  turn  itself  on  and  off  taxes  the  operator  with  keeping  up  with  what  the 
system  is  doing  unless  it  provides  the  operator  with  a  clear  indication  of  the  current  status. 

Research  has  also  explored  how  much  and  when  to  use  automation  (Kaber,  1996;  Kaber  & 
Endsley,  1997).  Kaber  and  Endsley  found  that  level  of  automation  had  a  large  effect  on  operator 
S  A,  however,  AA  had  an  effect  primarily  on  workload.  Kaber  found  a  significant  interaction 
between  AA  and  level  of  automation.  The  best  strategy  was  when  the  human  operator  performed 
at  a  level  of  automation  that  allowed  human  strategizing  and  computer  implementation  (level  of 
automation  3  -  Batch  Processing)  during  high  automation  cycle  times  (i.e.,  infrequent  human 
manual  control  through  AA).  This  combination  was  better  than  either  fully  automated 
performance  or  AA  cycles  that  had  high  human-control  times. 

Norman  (1989)  cited  fundamental  changes  in  the  amount  and  type  of  feedback  provided  by 
automated  systems  as  crucial.  We  need  to  determine  the  necessary  information  to  convey  to  the 
ATCS.  For  example,  when  the  FAA  automates  flight  strips,  there  may  be  loss  of  task  load 
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information  associated  with  the  number  of  flight  strips  or  with  the  physical  manipulation  of  flight 
strips  (Garland,  Stein,  Blanchard,  &  Wise,  1992;  Gronlund,  1992;  Hopkin,  1991). 

1.1.3  Summary 

The  successful  implementation  of  automation  will  depend  on  the  ATCSs’  ability  to  acquire 
different  types  of  skills,  retain  less  frequently  used  skills,  and  adapt  to  different  workload  levels. 
The  increased  complexity  of  systems  and  the  changes  they  induce  in  the  degree  of  ATCS 
involvement  requires  human  factors  considerations  in  developing  an  effective  ATC  system. 

Future  ATC  systems  may  impose  a  variety  of  challenges  to  ATCSs  including  information 
overload,  non-integrated  data,  rapidly  changing  parameters,  and  various  forms  of  automation. 
Evaluating  the  degree  to  which  prospective  system  designs  actually  provide  benefits  to  ATCSs  is 
an  important  goal  for  ATC  system  evaluation. 

1.2  Objectives 

The  introduction  of  DSTs  into  ATC  will  change  ATCSs’  behaviors  and  work  procedures  to 
accommodate  these  new  tools.  For  example,  ATCSs  may  no  longer  rely  on  flight  strips,  change 
Data  (D-side)  ATCSs’  assistance  to  the  Radar  (R-side),  and  learn  how  to  best  use  the  DST  to 
meet  the  ATCSs’  primary  goal  of  safety  (i.e.,  moving  from  a  novice  DST  user  to  an  expert  DST 
user).  With  the  expected  increases  in  air  traffic,  ATCSs  will  control  higher  volumes  of  traffic  for 
longer  durations  and  at  more  complex  levels  (i.e.,  more  aircraft  that  require  more  control  actions 
and  may  request  user-preferred  routes).  The  purpose  of  the  current  study  was  to  explore  in-depth 
the  effects  of  levels  of  automation  and  traffic  task  load  or  volume  on  ATCSs  in  an  ATC  en  route 
environment.  It  is  important  to  investigate  the  effects  of  these  anticipated  changes  on  ATCSs’ 
behaviors  and  cognitive  functions.  Through  careful,  controlled  experimentation,  researchers 
gain  information  about  the  possible  consequences  from  expected  and  planned  changes  in  the 
ATC  environment.  The  results  obtained  from  experimentation  can  then  guide  development  of 
the  most  appropriate  and  effective  strategies  to  offset  any  detrimental  consequences  from  these 
changes. 

We  manipulated  the  level  of  automation  that  ATCSs  used  in  controlling  traffic.  This  included 
three  levels  ranging  from  no  automation  to  limited  automation  to  full  automation.  To  simulate 
the  expected  increases  in  air  traffic,  we  manipulated  task  load  based  on  the  volume  of  air  traffic 
that  ATCSs  controlled.  ATCSs  controlled  both  low  and  high  traffic  task  loads.  Further,  ATCSs 
rotated  between  the  R-side  and  D-side  positions. 

We  anticipated  changes  in  ATCSs’  behaviors  and  cognitive  performance  with  the  manipulation 
of  automation,  task  load,  and  position.  We  examined  their  behavior  and  cognitive  processing 
through  objective  and  subjective  performance  and  behavioral  measures.  These  measures 
examined  SA,  workload,  visual  scanning,  communications,  and  trust.  We  provide  background 
information,  the  results,  and  the  discussion  for  each  of  these  measures  under  their  respective 
headings  in  the  Data  Set  Specific  Analysis,  Results,  and  Discussion  section. 
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2.  METHOD 


In  this  study,  16  ATCSs  performed  en  route  ATC  simulations  at  three  levels  of  automation  (No 
Automation,  Limited  Automation,  and  Full  Automation)  and  two  task  load  levels  (High  and 
Low).  The  ATCSs  worked  in  teams  of  two,  consisting  of  an  R-  and  D-side. 

2.1  Participants 

Sixteen  ATCSs  from  Air  Route  Traffic  Control  Centers  (ARTCCs)  within  the  United  States 
served  as  voluntary  participants.  All  participants  were  current,  non-supervisory,  full-time 
ATCSs  and  DSR  certified.  None  of  the  participants  was  on  medical  waiver  or  in  a  staff  position 
at  the  time  of  the  experiment.  Eleven  participants  had  normal  vision  and  five  had  corrected-to- 
normal  vision.  The  oculometer  design  limitations  excluded  bifocals,  trifocals,  or  hard  contact 
lenses  but  allowed  ATCSs  to  wear  corrective  lenses  or  soft  contact  lenses,  if  necessary.  The 
mean  age  of  the  participants  was  38.7  years  (34-44).  They  had  actively  controlled  traffic  for  a 
mean  of  15.1  years  (9-18.5).  Six  participants  had  worked  at  more  than  one  facility  during  their 
ATC  career.  The  participants  worked  air  traffic  for  an  average  of  1 1 .5  months  in  the  preceding 
12  months.  None  of  the  ATCSs  had  previously  received  training  on  a  DST.  Using  a  10-point 
scale,  participants  rated  their  current  skill  level  as  an  8.7  (6-10)  and  their  motivation  to 
participate  in  the  study  as  a  6.1  (3-9). 

The  Institutional  Review  Board  of  the  William  J.  Hughes  Technical  Center  approved  the 
protocol.  The  ATCSs  gave  their  written  consent  to  participate  in  the  experiment  (Appendices  A 
and  B  contain  the  participant  recruiting  form  and  the  Informed  Consent  Form).  The  research 
team  ensured  them  that  their  data  would  be  completely  confidential. 

2.2  Experimental  Staff 

A  research  team  composed  of  three  Human  Factors  Engineers  (HFEs)  and  two  ATC  Subject 
Matter  Experts  (SMEs)  conducted  the  study.  In  preparation  for  the  experiment,  the  HFEs 
designed  the  study,  questionnaires,  and  procedures.  The  ATC  SMEs  designed  the  scenarios. 
After  experiment  completion,  the  HFEs  performed  the  data  analyses  and  wrote  the  final  technical 
report.  During  the  first  week  of  training,  one  HFE  and  an  ATC  SME  explained  the  study  and  the 
informed  consent  policy  and  trained  the  participants  on  the  DST  and  the  airspace. 

For  the  experimental  portion  of  the  study,  two  HFEs  and  an  ATC  SME  conducted  the 
simulations.  The  HFEs  managed  the  experiment,  collected  the  data,  and  directed  support  staff. 
The  ATC  SME  conducted  the  Over-The-Shoulder  (OTS)  ratings  and  completed  SA  Verification 
ANalysis  Tool  (SAVANT)  forms  with  correct  answers.  Two  system  operators  brought  the  Host 
and  DST  systems  up  for  each  simulation  run,  while  another  operator  selected  the  appropriate 
scenario  and  prepared  the  Target  Generation  Facility  (TGF)  for  each  run.  The  study  used  six 
simulation  pilots  and  one  ghost  ATCS.  To  allow  rotation,  researchers  trained  nine  simulation 
pilots  using  procedures  from  past  experiments.  Support  engineers  ensured  that  the  hardware  and 
software  functioned  properly.  Clerical  staff  assisted  in  preparing,  copying,  and  distributing 
forms  and  questionnaires,  in  addition  to  preparing  data  sets,  means  and  standard  deviations 
(SDs),  Multivariate  Analysis  of  Variance  (MANOVA),  and  Analysis  of  Variance  (ANOVA) 
tables. 
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2.3  Materials 


In  the  next  sections,  we  describe  the  simulated  airspace  and  scenario  materials. 

2.3.1  Airspace 

The  airspace  used  in  this  study  was  a  generic  en  route  ARTCC  to  make  our  findings  easier  to 
generalize  and  to  increase  the  size  of  our  participant  pool  (Guttman,  Stein,  &  Gromelski,  1995). 
We  created  an  airspace  several  sectors  wide  because  the  DST  required  a  20-minute  look-ahead 
time.  Only  one  of  the  sectors  was  active  whereas  the  other  sectors  functioned  as  simulated 
ATCS  sectors  (i.e.,  ghosts).  During  the  simulation,  the  weather  conditions  required  instrument 
flight  rules  to  be  in  effect.  Figure  1  shows  and  Appendix  C  includes  pictures  of  the  airspace  used 
in  this  study,  called  Genera  Center.  Within  Genera  Center,  we  used  a  high  altitude  sector  200  x 
125  nms  with  boundaries  from  flight  level  (FL)  240  and  above.  Traffic  flow  consisted  of  arrivals 
handed  off  to  intermediate  sectors,  departures  climbing  from  intermediate  sectors,  and 
overflights  through  the  sector. 

For  the  staffing  of  Genera  Center,  ATCS  participants  controlled  one  sector  (Sector  07  depicted  in 
white  in  the  center  of  Figure  1),  while  we  staffed  all  adjacent  sectors  (depicted  in  gray  in  Figure 
1)  with  one  of  our  experimental  staff.  The  R-side  ATCS  ensured  separation,  initiated  control 
instructions,  and  operated  radios.  The  D-side  ATCS  scanned  the  radar  display  for  information 
and  assisted  the  R-side  ATCS  when  needed.  The  D-side  ATCS  also  ensured  separation,  but  on  a 
separate  position  that  shares  the  radar  display  with  the  R-side  ATCS.  The  D-side  ATCS 
operated  the  LL  phones  to  adjacent  sectors  and  facilities,  accepted  and  initiated  handoffs  for  the 
continued  smooth  operation  of  the  sector,  and  ensured  that  the  R-side  was  made  immediately 
aware  of  any  action  taken.  When  working  under  a  condition  that  included  the  DST,  the  D-side 
ATCS  monitored  the  DST  and  integrated  the  information  into  the  team’s  sector  plan.  The  ghost 
accepted  handoffs  from  our  participant  ATCSs  and  interacted  with  them  when  they  requested 
control  instructions  from  adjacent  sectors.  The  ghosts  confirmed  the  requested  control 
instructions,  contacted  the  simulated  aircraft,  and  instructed  the  simulation  pilots  to  fly  the 
aircraft  to  comply  with  the  ATCS  request. 

2.3.2  Scenarios 


The  experiment  consisted  of  1 8  different  scenarios.  We  designed  the  scenarios  with  multiple 
conflicting  traffic  routes  to  necessitate  the  use  of  the  DST  for  conflict  resolution.  Six  scenarios 
contained  moderate  task  loads  for  training  sessions,  and  we  used  six  low  and  six  high  task  load 
scenarios  for  experimental  sessions.  Each  training  scenario  lasted  30  minutes,  whereas  the 
experimental  scenarios  ranged  in  duration  from  33  to  40  minutes  to  enable  measures  of  SA  with 
the  SA  Global  Assessment  Technique  (SAGAT).  We  randomized  the  presentation  order  of  the 
scenarios  and  rotated  the  scenarios  through  each  automation  level. 
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Figure  1 .  Illustration  of  Genera  Center,  Sector  7. 

2.4  Location  and  Equipment 

The  following  subsections  describe  the  location  where  the  study  took  place  and  the  hardware  and 
software  we  used  during  the  study. 

2.4.1  Location  and  Training  and  Experimental  Setups 

The  experiment  encompassed  three  separate  locations  and  three  different  workstation  platforms. 
Each  location  and  operating  system  allowed  for  DST  training,  airspace  training,  and 
experimental  sessions  of  the  study  to  be  conducted.  We  discuss  each  location  and  its  platform  in 
the  following  subsections. 

2.4.2  DST  Training  Platform 

Training  on  the  DST  and  familiarization  with  the  airspace,  the  Letters  of  Agreement  (LOAs),  and 
Standard  Operating  Procedures  (SOPs)  took  place  in  a  classroom  in  the  RDHFL.  The  training 
hardware  consisted  of  an  instructor  position  and  two  student  positions.  These  workstations 
displayed  stand-alone  versions  of  the  DST.  The  DST  itself  ran  on  a  DEC  Alpha  system  (Digital 
Equipment  Corporation,  Boston,  MA)  in  the  I2F  with  remote  displays  on  the  instructor  and 
student  positions  that  consisted  of  SUN  (SUN,  Palo  Alto,  CA)  workstations  with  19  in.  displays, 
a  keyboard,  and  a  three-button  mouse. 

2.4.3  Airspace  Training  Platform 

For  training  on  the  airspace,  the  RDHFL  provided  a  high  fidelity  ATC  simulation  environment. 
We  used  two  stand-alone  ATCoach  simulators  (UFA,  1998),  running  on  SUN  machines.  The 
ATCS  stations  included  a  radar  display,  full  flight  strip  bay,  an  ARTCC  keyboard,  and  a 
trackball.  A  high-resolution  (2,000  by  2,000  pixel)  monitor  (Sony  Corporation,  Japan)  displayed 
the  radar  display.  Three  PC-based  simulation  pilot  workstations  running  on  SUN  operating 
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systems  were  connected  to  each  ATCoach  (UFA)  simulator  to  allow  simulated  pilot  participation 
in  the  scenarios.  The  ATCoach  (UFA)  simulators  allowed  training  of  each  participant  on  the 
airspace  independently  and  simultaneously. 

2.4.4  Experimental  Platform 

The  experimental  sessions  took  place  in  the  en  route  lab  in  the  I2F.  The  I2F  provides  areas  and 
equipment  for  evaluation  of  ATC  infrastructure  enhancements  and  subsystems  and  integration 
and  interoperation  of  ATC  subsystems  and  functions  and  is  configurable  to  support  multiple 
users  simultaneously. 

For  the  experimental  sessions,  we  used  an  integrated  system  including  the  TGF,  the  Host 
computer,  and  a  full  DSR  workstation  with  all  functions  normally  expected  in  an  operational 
setting,  and  the  DST.  We  used  the  TGF  to  generate  targets  and  air  space.  A  2,000  by  2,000 
pixel,  29  in.  video  display  unit  (Sony)  presented  the  radar  information.  An  Air  Traffic  Workload 
Input  Technique  (ATWIT)  (Stein,  1985)  was  mounted  immediately  next  to  the  DSR  display 
within  easy  reach  of  the  participant  for  input  of  workload  ratings.  The  workstation  had  a  DSR 
flight  strip  bay,  an  en  route  keyboard,  and  a  trackball  with  three  buttons.  An  LL  allowed 
interfacility  and  intrafacility  communications.  The  right  trackball  button  served  as  a  home  key 
that  returned  the  cursor  to  the  center  of  the  DSR  when  pressed.  The  D-side  ATCS  had  access  to 
the  DST.  Different  from  the  configuration  in  the  field,  this  DST  ran  on  a  29  in.  video  display.  A 
network  linked  six  simulation  pilot  displays  with  the  R-  and  D-side  positions.  Each  simulation 
pilot  station  allowed  control  of  several  aircraft. 

2.4.4. 1  DST 

The  DST  includes  several  display  windows  that  capture  various  flight  data  and  conflict 
information.  One  display  window  presents  an  aircraft  list  of  all  aircraft  inbound  to  the  sector 
(Figure  2).  The  list  shows  the  controlling  sector  of  a  particular  aircraft;  flight  data  such  as  flight 
route,  aircraft  type,  speed,  altitude  and  beacon  code;  and  conflict  indication. 

The  conflict  indication  depicts  red  for  predicted  violations  of  less  than  5  nms  and  yellow  when 
less  than  12  nms  but  more  than  5  nms.  As  shown  in  Figure  2,  TWA483  (the  highlighted  line) 
has  a  red  conflict  indicated,  but  the  other  aircraft(s)  with  which  it  is  in  conflict  are  not  identified. 

The  graphic  plan  display  window  depicts  aircraft  and  resembles  the  DSR  display  (Figure  3).  It 
can  show  all  conflicts,  conflicts  only  in  the  ATCSs’  sector,  or  conflicts  of  a  specific  type  (e.g., 
red).  As  shown  in  Figure  3,  DAL317  and  DAL540  have  a  predicted  conflict  that  is  clearly 
shown  on  the  graphic  plans  display. 
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Figure  2.  Aircraft  list  display  window  from  DST. 


Figure  3.  The  graphic  plans  display  window  from  the  DST. 

In  Appendix  D,  a  plans  display  textually  depicts  the  computer  and  aircraft  identifications,  aircraft 
altitude,  speed,  and  route  along  with  any  problem  information  (Figure  D-l).  The  departure  list 
shows  flight  information  for  proposed  departure  flights.  It  includes  aircraft  and  computer 
identifications,  departure  time,  aircraft  type  and  equipment,  filed  altitude,  beacon  code,  and  route 
of  flight.  These  plans  are  not  probed  for  conflicts  (Figure  D-2).  A  response  display  shows 
system  messages  describing  problems  from  an  ATCS  action  (Figure  D-3). 


Trial  flight  planning  allows  the  ATCS  to  enter  in  a  revised  altitude,  speed,  or  route  change  to 
resolve  a  potential  conflict.  A  trial  plan  can  be  created  from  the  aircraft  list,  departure  list,  or 
graphic  plans  display.  The  DST  then  shows  whether  the  change  would  no  longer  result  in  a 
conflict.  This  can  be  presented  textually  or  graphically. 

The  DST  also  offers  the  capability  to  coordinate  trial  plans  between  sectors.  However,  the  DST 
configuration  used  in  the  study  did  not  allow  coordination  between  sectors. 
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2,4. 4. 2  Video  Camera  and  Video  Tape  Configuration 

We  taped  the  video  images  of  both  ATCS  positions  (Figure  4,  top  and  side  views  of  both  D-side 
and  R-side  ATCSs)  and  streamed  audio  and  video  to  disk.  We  copied  the  MPEG  files  onto  a 
CD-ROM  for  further  analysis. 


Figure  4.  Example  of  top  and  side  views  of  ATCSs. 

To  record  the  intra-team  communications,  we  developed  a  program,  Orasis  (Dauphin 
Technology,  1999),  that  ran  on  a  portable  device  when  conducting  live  coding.  The  software  has 
a  layout  as  depicted  in  Figure  5.  The  left  column  of  buttons  of  the  software  indicated  the 
message  content.  The  second  column  indicated  the  message  type.  Finally,  the  third  column 
indicated  message  modality.  When  conducting  post  hoc  coding,  we  used  a  behavioral  coding 
software  program,  Observer  Video-Pro  (Noldus,  2000).  The  software  configuration  used  the& 
Controller-to-Controller  Communications/Coordination  Taxonomy  (C4T)  coding  scheme 
developed  by  Peterson,  Bailey,  &  Willems  (in  press).  This  allowed  an  SME  to  observe  side  and 
top  views  of  both  the  R-  and  D-side  and  code  behaviors  in  real-time.  Using  this  matrix,  we 
captured  specific  verbal  and  non-verbal  communication  behaviors  as  well  as  the  sequence  and 
timing  of  the  behaviors. 


2.4.4.3  Communications  Configuration 

We  used  communication  systems  that  are  functionally  similar  to  those  in  use  in  the  RDHFL. 
There  were  communication  links  between  the  ATC  SME,  OTS  observer,  simulation  pilots, 
experimenters,  and  push-to-talk  (PTT)  recording.  The  equipment  monitored  communications 
and  recorded  times  and  frequencies. 
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Figure  5.  Coding  software  interface. 

2.4.4.4  Situation  Awareness  Verification  Analysis  Tool 

SAVANT  ran  on  separate  multiple  network  SUN  workstations  programmed  in  C++,  Java,  and 
ODS  Toolbox  (Orthogon,  1999),  one  for  the  D-side  and  one  for  the  R-side  ATCSs.  Twenty 
seconds  after  ATWIT  alerted  the  participants  for  a  workload  rating,  both  the  R-side  and  D-side 
ATCSs’  DSR  monitors  displayed  the  SAVANT  query  for  3  seconds;  after  that,  the  replication  of 
the  DSR  screen  appeared  for  9  seconds  while  they  responded  to  the  query.  For  more  detailed 
information  on  the  implementation  of  SAVANT,  see  Appendix  E. 

The  SAVANT  display  closely  matched  the  DSR  display.  We  presented  the  ATCS  with  several 
queries.  The  queries  consisted  of  both  Aircraft-Pair  and  Sector-Based  questions.  Aircraft-Pair 
questions  were  relational;  they  always  asked  about  a  pair  of  aircraft.  The  query  asked  about  the 
relationship  between  one  aircraft  and  either  another  aircraft  or  airspace  for  the  current  situation 
or  a  future  situation.  The  questions  assessed  the  comprehension  and  projection  components  of 
SA.  All  visual  information  was  present  except  for  the  information  about  which  the  query  was 
asking  (Figure  6).  It  was  a  forced  choice  question.  In  contrast,  Sector-Based  questions  were 
open-ended  with  multiple  responses  possible  and  assessed  the  ATCSs  comprehension  of  the 
current  situation  only.  The  only  information  available  for  the  question  was  the  aircraft  position 
(Figure  7).  We  recorded  the  ATCS  responses  by  having  them  click  on  an  aircraft. 

2.4.4. 5  Situation  Awareness  Global  Assessment  Technique 

In  addition  to  the  SAVANT  tool,  we  used  the  SAGAT  to  measure  SA  at  the  end  of  each 
scenario.  SAGAT  ran  on  Apple  PowerBooks  programmed  in  HyperCard.  Section  3.1.2  provides 
detailed  descriptions  of  the  background,  data  reduction,  analysis,  and  discussion  of  the  SAGAT 
data. 
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Figure  6.  Comparison  of  DSR  and  SAVANT  aircraft  representation  for  aircraft-pair  SA  queries. 


k: 

TWA234 
/  230  208 
/  103  380 

& 

AAL879 

210C 

142  420 

ssnV 

DSR 

Sector-based 

Queries 

Figure  7.  Comparison  of  DSR  and  SAVANT  aircraft  representation  for  sector-based  SA  queries. 

2.4.4.6  Air  Traffic  Workload  Input  Technique 

An  ATWIT  device  (Stein,  1985)  recorded  response  latencies  (e.g.,  times  to  respond)  and 
workload  ratings  during  all  conditions.  The  participants  made  a  rating  on  the  ATWIT  device 
every  3  minutes.  Before  each  rating,  a  tone  alerted  the  participant  who  then  had  20  seconds  to 
make  a  workload  rating.  The  participants  used  a  scale  of  1  (low  workload)  to  10  (high  workload) 
(Appendix  F  contains  the  detailed  ATWIT  instructions).  ATWIT  is  a  reliable  and  relatively 
unobtrusive  real-time,  on-line  measure  of  subjective  workload. 

2.4.4.7  Oculometer 


We  used  an  oculometer  (Applied  Science  Laboratories,  1991)  consisting  of  an  eye  and  head 
tracking  system.  This  system  recorded  the  Point-of-Gaze  (POG)  and  pupil  diameter  of  a  person 
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by  using  near  infrared  reflection  outlines  from  the  pupil  and  cornea.  For  an  extensive  description 
of  both  the  hardware  and  the  software  used  for  eye  tracking,  we  refer  the  reader  to  previous 
reports  (Willems,  Allen,  &  Stein,  1999;  Willems  &  Truitt,  1999).  Willems  et  al.  (1999) 
indicated  that  the  exposure  to  the  infrared  illumination  while  wearing  the  oculometer  is  less  than 
4%  of  the  intensity  of  that  when  walking  outside  on  a  sunny  day. 

To  enable  accurate  calculation  of  the  location  of  the  POG,  we  determined  the  exact  three- 
dimensional  location  of  several  surfaces  (or  scene  planes)  relative  to  the  oculometer  coordinate 
system.  The  procedures  used  for  this  initial  calibration  process  measure  distances  of  known 
points  on  the  scene  planes  and  determine  the  coordinates  of  each  of  these  points  relative  to  the 
oculometer  three-dimensional  coordinate  system.  The  oculometer  then  used  the  position  and 
orientation  of  the  scene  planes  to  determine  the  local  coordinates  (i.e.,  the  coordinates  relative  to 
a  two-dimensional  coordinate  system  attached  to  each  of  the  scene  planes). 

Once  the  oculometer  software  stored  the  exact  position  of  the  scene  planes,  one  only  needs  a 
participant  calibration  before  each  of  the  simulations  to  correct  for  the  way  the  head-mounted 
magnetic  head  tracker  and  optical  eye  tracker  fit  on  the  participant’s  head  and  for  distortions  in 
the  optical  system.  We  used  a  17-point  calibration  grid  displayed  on  2000  x  2000  display  similar 
to  that  depicted  in  Figure  8.  During  this  final  calibration,  we  instructed  the  participant  to  sit  still 
and  to  focus  his  or  her  gaze  on  the  numbered  points  as  we  called  them  out.  We  used  the 
oculometer  software  to  automatically  enter  the  participant’s  POG  for  each  of  the  17  points.  The 
software  then  used  the  known  locations  of  these  points  to  determine  the  adjustments  it  needed  to 
make  to  fit  POG  to  the  exact  location  of  the  calibration  points.  At  the  end  of  the  calibration 
procedure,  the  experimenter  verified  that  the  participant’s  POG  coincided  with  the  system’s 
coordinates  by  having  the  participant  look  at  several  points  of  the  calibration  grid. 


Figure  8.  Example  of  the  calibration  screen  used  with  the  oculometer. 
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2.5  Design 


Our  study  was  a  2  x  2  x  3  design  and  contained  the  following  IVs:  three  levels  of  automation, 
two  levels  of  task  load,  and  two  levels  of  position.  The  DVs  consisted  of  several  measures  to 
assess  ATCSs’  performance  and  behavior.  We  collected  scores  on  all  measures  for  all  trials  and 
compared  them  for  each  level  of  the  experimental  variables. 

2.5.1  Automation 


This  study  used  three  levels  of  automation.  The  no  automation  (baseline  or  first  level)  featured 
paper  flight  strips.  The  limited  automation  (second  level)  featured  electronic  flight  strips  with 
conflict  indication  but  no  trial  flight  planning  (aircraft  list  from  DST).  The  full  automation  or 
(third  level)  featured  electronic  flight  strips  with  flight  planning  (aircraft  list,  graphic  plan 
display,  and  trial  plan  display  from  DST). 

2.5.2  Task  load 


We  defined  high  and  low  task  load  as  the  difficulty  of  the  scenario,  operationalized  by  the 
number  of  aircraft  and  the  complexity  of  the  traffic  situation.  For  this  experiment,  we  defined  a 
low  task  load  scenario  as  having  a  level  of  complexity  at  which  a  first  line  supervisor  would  be 
about  to  remove  the  assistance  of  a  D-side  ATCS.  We  defined  a  high  task  load  scenario  as 
having  a  level  of  complexity  at  which  a  first  line  supervisor  would  be  about  to  assign  the  extra 
assistance  of  tracker  (i.e.,  move  from  a  two-  person  team  to  a  three-person  team).  Figure  9 
shows  the  average  number  of  aircraft  for  each  3 -minute  interval  that  ATCSs  controlled  traffic  in 
both  the  low  and  high  task  load  scenarios.  The  dashed  lines  represent  the  variability  among  the 
scenarios  within  each  task  load  level. 
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Figure  9.  Average  number  of  aircraft  for  low  and  high  task  load  scenarios. 

2.5.3  Position 


We  used  two  levels  of  position:  R-side  and  D-side.  Appendix  G  contains  complete  descriptions 
for  R-side  and  D-Side  roles  and  responsibilities. 
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We  calculated  summary  DVs  for  scenario  and  interval.  After  each  scenario,  we  collected  Post- 
Scenario  Questionnaire  (PSQ),  trust.  National  Aeronautics  and  Space  Agency  (NASA)  Task 
Load  Index  (TLX),  OTS,  and  SAGAT  ratings  (Appendix  H  to  Appendix  J)  and  visual  scanning 
and  performance  summary  data.  At  intervals,  we  collected  ATWIT  and  SAVANT  ratings. 
Participants  completed  an  entry  questionnaire  and,  at  the  end  of  the  study,  an  exit  questionnaire 
(Appendix  K  and  Appendix  L,  respectively).  We  used  all  questionnaire  data  in  its  raw  form. 

Table  3  summarizes  the  data  sets  we  collected  during  the  study.  Each  of  the  data  sets  provides 
insight  into  ATCS  performance  and  behavior. 

Table  3.  The  Data  Sets  Recorded  during  the  Experimental  Scenarios 

SA  via  SAVANT 

•  SA  via  SAGAT 

•  Workload  via  the  ATWIT  device 

•  Eye  tracking  of  the  D-side  ATCS  at  60  samples  per  second 

•  Continuous  recording  of  communications  between  the  R-  and  D- 
sides 

•  Recorded  communications  between  ATCSs  and  simulation  pilots  on 
MPEG  files,  with  a  time  stamp 

•  Unix-based  PTT  software  identifying  the  speaker,  and  at  what  time 
and  for  how  long  the  speaker  keyed  the  microphone 

•  Aircraft  data  and  pilot/ATCS  entries  into  the  system  (TGF) 

•  System  Analysis  Recording  (SAR)  data  (Host) 

•  Questionnaires _ _ 

2.6  Procedure 


ATCSs  participated  in  the  experiment  for  2  consecutive  work  weeks.  The  first  week  consisted  of 
familiarization  and  training,  and  the  second  week  consisted  of  experimental  sessions.  The 
morning  of  the  participants’  first  day  consisted  of  a  briefing  and  a  familiarization  period.  The 
HFEs  explained  the  experiment,  the  oculometer,  differences  between  TGF  and  their  own 
equipment,  and  the  confidentiality  of  their  identity.  Researchers  provided  an  informed  consent 
briefing  and  assurance  that  participation  was  voluntary.  Participants  signed  an  informed  consent. 
The  ATCSs  then  completed  an  Entry  Questionnaire  that  included  demographic  questions  about 
age,  experience  level,  need  for  corrective  glasses,  and  so  on. 

The  decision  support  automation  training  consisted  of  2  days  of  classroom  instruction  and  one 
day  of  hands-on  training.  After  instructing  the  ATCSs  about  the  LOAs  and  the  SOPs,  the 
ATCSs  received  background  information  on  the  DST.  This  included  the  functionality,  human- 
computer  interface,  and  operation  of  the  DST.  The  ATCSs  then  received  hands-on  training  with 
the  DST.  We  replayed  interactive  scenarios  during  training.  The  next  day,  we  briefed 
participants  on  the  airspace,  and  they  worked  simulations  to  learn  the  airspace.  The  airspace 
modeled  for  the  training  sessions  was  identical  to  the  airspace  used  during  the  experimental 
sessions.  Our  ATC  SME  trained  each  participant  in  the  use  of  the  airspace,  scenario  flow,  and 
traffic  type. 

The  first  day  of  the  second  week  consisted  of  hands-on  training  with  all  experimental  equipment 
and  procedures  in  place.  Before  beginning  experimental  trials,  we  trained  each  participant  in  the 
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use  of  equipment,  including  the  DSR  workstation,  communications,  the  oculometer,  ATWIT, 
SAVANT,  and  SAGAT.  At  the  end  of  training,  participants  mastered  the  airspace  and  all  of  the 
equipment  used  in  the  experiment.  Finally,  the  last  3  days  consisted  of  experimental  scenarios. 
ATCSs  had  a  1 5  to  20  minute  break  between  trials  and  60  minutes  for  lunch.  Appendix  M 
presents  a  detailed  schedule  of  activities. 

3.  RESULTS 


We  measured  participant  performance  and  behavior  across  five  constructs:  SA,  workload,  visual 
scanning,  trust,  and  performance.  To  keep  the  background,  results,  and  discussion  for  a  specific 
data  set  in  close  proximity  to  one  another,  we  report  them  per  construct. 

We  categorized  the  data  sets  into  three  groups  based  on  collection  method:  those  collected  (1)  by 
scenario,  (2)  by  interval,  and  (3)  continuously  (see  Table  4).  Data  collected  after  each  scenario 
include  SAGAT  and  questionnaire  ratings.  Data  collected  by  interval  include  ATWIT  and 
SAVANT.  Data  collected  continuously  include  visual  scanning,  communications,  system 
entries,  and  System  Analysis  Recording  (SAR)  tapes.  For  analysis  purposes,  we  calculated 
summary  statistics  on  continuous  data  per  interval  and  per  scenario.  We  summarized  the  visual 
scanning  data,  the  R-  and  D-side  communications,  ATCS/pjlot  communications,  and  pilot/ATCS 
system  entries  by  scenario.  We  also  calculated  pilot/ATCS  system  entries  by  interval. 


Table  4.  Scenario,  Interval,  and  Continuous  Data  Sets 


Scenario  Data 

Interval  Data 

Continuous  Data 

•  SA  via  SAGAT 

•  Questionnaires 

•  SA  via  SAVANT 

•  Responses  to  the 
ATWIT  device 

•  Eye  tracking  of  the  D-side  ATCS  at  60  samples  per  second 

•  Continuous  recording  of  communications  between  the  R- 
and  D-sides 

•  ATCS/pilot  communications 

•  Aircraft  data  and  pilot/ATCS  entries  into  the  system 

•  SAR  data 

Due  to  problems  in  initial  counterbalancing,  the  first  10  participants  did  not  experience  each 
condition  of  the  design.  We  replaced  any  missing  data  with  the  overall  mean.  The  remaining  six 
participants  completed  all  condition  levels.  We  performed  analyses  using  their  responses  as  a 
second  data  set.  With  the  analysis  of  the  second  data  set,  we  examined  whether  the  replacement 
for  missing  data  had  removed  effects  of  experimental  conditions. 

Unless  otherwise  noted,  all  results  represent  the  findings  for  the  data  sets  of  16  participants. 
When  the  results  of  the  smaller,  six  participant  data  sets  differed  from  the  1 6  participant  data 
sets,  we  present  these  findings  separately. 

For  a  description  of  general  statistical  methods  as  well  as  detailed  information  about  the 
statistical  methods  used  in  this  study,  see  Appendix  N.  We  computed  MANOVAs  to  compare 
effects  on  multiple  variables  and  ANOVAs  for  effects  on  single  DV.  We  tested  the  Wilks’  A 
statistic  using  a  level  of p<  .05  and  report  the  equivalent  F  statistic.  We  report  the  most 
commonly  used  alpha  level  closest  to  the  actual  p  value  obtained.  If  the  results  of  the  MANOVA 
were  statistically  significant  (p  <  .05),  we  performed  univariate  ANOVAs  to  determine  which  of 
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the  DVs  were  significantly  different  across  experimental  conditions.  We  based  the  significance 
of  an  ANOVA  result  on  an  adjusted  alpha  level  using  the  following  formula: 

alpha  (overall)  =  l-(l-alpha(individual))n  where  n  is  the  number  of  variables 
or: 

alpha  (individual)  =  l-(l-alpha(overall))1/n 

We  report  the  adjusted  alpha  level  with  each  analysis.  If  the  result  of  an  ANOVA  was 
statistically  significant,  we  performed  appropriate  post  hoc  tests  to  determine  which  conditions 
were  responsible  for  the  significance. 

Other  researchers  have  used  a  more  lenient  approach  when  investigating  the  effects  of 
manipulation  on  DVs  by  not  adjusting  the  alpha  level.  Such  an  approach  may  inflate  the  overall 
alpha  level  but  allows  researchers  to  investigate  trends  in  the  data.  In  the  current  study,  we 
follow  such  an  approach  to  investigate  trends  (Table  5).  We  use  the  term  “trend”  to  indicate  a 
primary  trend.  A  primary  trend  indicates  an  effect  that  did  reach  significance  at  the  multivariate 
level  but  had  a p-value  at  the  univariate  level  greater  than  the  adjusted  alpha  but  lower  than  .05. 
It  also  could  indicate  that  it  did  not  reach  significance  at  the  multivariate  level  but  had  a  p- value 
less  than  the  adjusted  alpha  at  the  univariate  level.  A  secondary  trend  refers  to  an  effect  that  did 
not  reach  significance  at  the  multivariate  level  but  was  higher  than  an  adjusted  alpha  but  lower 
than  .05  at  the  univariate  level. 


Table  5.  Types  of  Trends 


Trend 

Multivariate 

Univariate  /7-value 

Primary 

Significant 

<  .05,  >  adjusted  alpha 

Primary 

Not  significant 

<  adjusted  alpha 

Secondary 

Not  significant 

<  .05,  >  adjusted  alpha 

In  the  figures  graphically  representing  the  results,  we  provide  means  and  SDs.  The  SDs  indicate 
the  between-subject  variance. 

3.1  Situation  Awareness 

ATCSs  work  in  a  dynamic  environment  in  which  they  are  continually  assessing  changing  aircraft 
information  in  order  to  maintain  aircraft  separation  while  moving  aircraft  efficiently  through  the 
airspace.  They  must  maintain  “the  picture”  to  fulfill  these  goals.  As  outlined  by  Endsley  and 
Smolensky  (1998),  ATCSs  base  decisions  on  their  mental  model  of  the  situation  and  refer  to  this 
as  “the  picture.”  In  turn,  what  ATCSs  refer  to  as  the  picture  is  what  researchers  investigate  as 
SA  (Endsley  &  Smolensky).  Hence,  the  ATCS’s  ability  to  perform  traffic  management 
functions  directly  hinges  on  possessing  comprehensive  SA. 

There  is  currently  no  agreed  upon  definition  that  captures  the  essence  of  SA  (Endsley,  1989a,  b; 
Fracker,  1989;  Mogford  &  Tansley,  1991).  Endsley’s  definition  of  SA  is  “...the  perception  of  the 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their 
meaning,  and  the  projection  of  their  status  in  the  near  future”  (Endsley,  1988a,  p.  3).  Most 
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researchers  agree  that  the  perception  and  understanding  of  elements  in  the  present  situation  and 
the  use  of  this  information  to  anticipate  future  events  are  important  in  maintaining  SA. 

SA  is  a  cognitive  construct  separate  from  workload,  performance,  and  decision  making  that 
highlights  the  importance  of  future  perceptions  and  the  anticipation  of  future  events  within  a 
goal-directed  dynamic  environment  (Durso  &  Gronlund,  1999).  It  forms  input  to  decision¬ 
making  and  actions  and  is  important  for  successful  performance,  and  it  has  both  temporal  and 
spatial  components  that  reflect  the  current  goals  of  the  ATCS  (Endsley  &  Smolensky,  1998). 

SA  has  been  cited  as  crucial  for  effectively  monitoring  an  automated  system  (Endsley  & 
Smolensky,  1998)  and,  at  decreasing  levels,  contributing  to  more  severe  operational  errors  (OEs) 
in  ATC  (Durso,  Truitt,  Hackworth,  &  Cruthfield,  1998;  Rodgers  &  Nye,  1993).  Its  relationship 
with  workload  is  more  complex;  high  workload  may  lead  to  decreases  in  SA,  whereas,  decreases 
in  SA  may  lead  to  higher  workload  levels  (Endsley  &  Smolensky). 

In  the  case  of  ATC,  the  R-side  and  D-side  ATCSs  form  a  team.  Within  this  team,  each  person 
has  roles  and  responsibilities.  Task  allocation  defines  the  SA  requirements  of  each  team 
member.  However,  the  team  members  have  a  need  for  a  considerable  amount  of  overlapping 
information.  These  are  their  shared  SA  requirements.  The  team  can  achieve  effective  joint 
performance  when  each  member  has  the  SA  needed  for  his  or  her  role,  and  the  team  members 
have  the  shared  SA  required  for  effective  team  coordination.  Both  R-side  and  D-side  ATCSs 
need  to  be  aware  of  the  state  of  the  aircraft  within  the  airspace,  but  R-side  ATCSs  also  need  to  be 
aware  of  the  more  action-oriented  items  of  SA  such  as  an  aircraft  conformance  to  clearances. 

Measurement  of  SA 


The  process  model  developed  by  Endsley  (1996)  shown  in  Figure  10  depicts  the  issues  that  are 
involved  in  selecting  measures  of  SA.  This  model  shows  the  stages  involved  in  the  perception- 
action  sequence.  We  have  shown  these  closely  coupled  stages  as  separate  for  simplicity  only. 
Moderating  factors  that  may  influence  each  stage  appear  on  the  left  side.  On  the  right,  classes  of 
measures  that  are  appropriate  to  each  stage  appear.  Some  of  these  measures  will  be  indirect 
indices  of  SA  and  others  will  be  more  direct. 

Researchers  have  used  many  different  methods  to  measure  how  operators  develop  and  maintain 
SA.  Previously  used  measures  include  physiological  measures  such  as  eye  movements  (Moray 
&  Rotenberg,  1989;  Wierwille  &  Eggemeier,  1993),  verbal  protocol  analysis  (Ohnemus  &  Biers, 
1993;  Sullivan  &  Blackman,  1991),  retrospective  recall  (DeGroot,  1995;  Kibbe,  1988),  rating 
techniques  (Reid  &  Nygren,  1988;  Taylor,  1990),  memory  probes  (Endsley,  1988a,  b),’and  on¬ 
line  queries  (Durso  et  ah,  1 995).  Most  of  these  techniques  have  demonstrated  some  degree  of 
validity  and  usefulness. 

Vidulich  (in  press)  completed  an  extensive  meta-analysis  of  SA  measures.  This  analysis 
provides  an  interesting  comparison  of  the  sensitivity  of  the  various  techniques  for  measuring  SA, 
but  the  study  does  not  address  the  issue  of  validity.  In  the  following  sections,  we  describe 
process  indices  and  direct  and  indirect  measures  of  SA. 
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MODERATING  FACTORS 


MEASURE 


Figure  10.  Measures  of  SA  (from  Endsley,  1996). 

Process  Indices 

Individuals’  characteristics  will  influence  assessment  processes  used  in  acquiring  information 
from  the  environment  (Endsley  &  Bolstad,  1994).  Differences  in  underlying  abilities  including 
spatial,  attentional,  memorial,  perceptual,  and  cognitive  contribute  to  this  finding.  Individuals 
will  also  form  strategies,  skills,  and  knowledge  with  experience  and  training  that  will  contribute 
to  their  SA. 

Process  measures  may  provide  an  indirect  indication  of  operator  SA.  Some  may  be  useful  in 
conjunction  with  each  other.  Process  tracing  tools  used  in  the  study  of  decision  making  may  be 
applicable  to  the  study  of  SA  processes.  Eye-trackers  may  help  uncover  how  operators  allocate 
attention  in  acquiring  SA  and  their  typical  scan  patterns.  This  information  may  provide  insights 
into  SA  acquisition  or  into  the  mental  models  directing  this  process.  Stein  (1992)  found  that  it 
takes  about  2-3  minutes  for  scan  patterns  to  stabilize  after  taking  control  of  a  sector.  Smolensky 
(1993)  found  similar  results  for  scan  patterns  and  saccades  that  negatively  correlated  with  a 
SAGAT  measure  of  SA. 


Studying  the  communications  process  between  operators  may  provide  useful  information  on  the 
types  of  information  that  are  lacking  from  displays,  verbal  techniques  used  for  acquiring  SA,  and 
differences  in  SA  strategies  between  individuals.  Verbal  protocols  may  help  uncover  what  the 
operator  attends  to  and  may  provide  insight  into  how  that  information  is  integrated.  Sullivan  and 
Blackman  (1991)  used  verbal  protocols  to  investigate  the  relationship  between  working  memory 
and  long-term  memory  in  maintaining  SA. 

These  techniques  provide  useful  information  on  SA  processes.  However,  because  verbal 
communications  and  verbal  protocols  take  place  in  a  limited  period,  they  may  not  be  complete 
representations  to  what  ATCSs  process  or  attend.  Eye-trackers  and  information  acquisition 
methods  may  capture  the  SA  acquisition  process. 

Sarter  and  Woods  (1991)  have  proposed  using  a  scenario  manipulation  method.  Here, 
researchers  alter  displayed  information  in  some  unpredicted  way.  Artificial  manipulation  during 
a  simulation  may  provide  useful  insights  into  the  SA  process.  Tenney,  Adams,  Pew,  Huggins, 
and  Rogers  (1992)  discuss  using  this  technique  to  lead  subjects  “down  the  garden  path”  thus 
investigating  factors  that  may  lead  directly  to  incorrect  or  missed  assessments  of  situations. 

Direct  Measures  of  SA 


The  subjective  assessment  of  SA  is  attractive  because  it  is  inexpensive  and  easy  to  administer. 
However,  ATCSs  may  not  know  about  their  own  inaccuracies  or  of  what  information  they  are 
unaware  (Schroeder  &  Nye,  1993).  However,  self-ratings  may  be  useful  for  providing  an 
assessment  of  operators’  degree  of  confidence  in  their  SA  (Endsley  &  Smolensky,  1998). 

One  of  the  best-known  subjective  scales  is  the  Situational  Awareness  Rating  Technique  (SART) 
developed  by  Taylor  (1990).  Participants  rate  a  system  design  based  on  the  amount  of  demand 
on  attentional  resources,  supply  of  attentional  resources,  and  understanding  of  the  situation 
provided.  Selcon  and  Taylor  (1990)  have  found  that  SART  correlated  with  performance 
measures.  Selcon,  Taylor,  and  Koritsas  (1991)  showed  SART  to  be  sensitive  to  changes  in  task 
demands,  correlating  with  the  NASA-TLX  measure  of  workload.  Crabtree,  Marcelo,  McCoy, 
and  Vidulich  (1993)  found  SART  sensitive  to  most  display  manipulations,  particularly  the 
attentional  demand  scale.  Researchers  have  compared  the  SART  to  the  SAGAT.  Endsley, 
Selcon,  Hardiman,  and  Croft  (1998)  found  no  correlation  between  the  SAGAT  scores  and  the 
SART.  However,  the  SART  highly  correlated  with  a  subjective  measure  of  performance  and  a 
subjective  measure  of  pilot  confidence  level. 

As  another  approach  to  developing  a  standardized  subjective  measure  of  SA,  Vidulich  and 
Hughes  (1991)  used  a  modified  version  of  the  Subjective  Workload  Dominance  (SWORD) 
technique  to  obtain  subjective  evaluations  of  the  SA  provided  by  displays.  SA-SWORD  has 
individuals  provide  a  comparative  preference  for  displays  on  a  9-point  scale  based  on  their 
beliefs  about  the  amount  of  SA  provided  by  each.  They  found  the  technique  discriminated 
between  two  display  formats  and  had  inter-rater  reliability. 

Outside  observers  can  also  assess  subjective  SA.  An  advantage  of  this  method  is  that  trained 
observers  may  have  a  more  complete  knowledge  of  reality  than  the  rater.  A  shortcoming  is  that 
observers  will  have  only  limited  knowledge  about  the  ATCS’s  concept  of  the  situation.  ATCS 


24 


actions  and  speech  may  provide  diagnostic  information  on  explicit  SA  problems  and  indicate  that 
the  ATCS  knows  certain  information,  supporting  observer  judgments.  Actions  and 
verbalizations  cannot  provide  a  complete  representation  of  ATCS  SA.  As  pointed  out  by  Durso 
and  Gronlund  (1999),  SA  is  a  cognitive  construct.  Outside  observers  cannot  know  what  the 
participant  knows  mentally.  Efforts  to  elicit  more  information  may  augment  natural 
verbalizations,  but  this  may  alter  the  participant’s  distribution  of  attention,  thus  altering  SA. 

Questionnaires  collect  detailed  information  about  ATCS  perceptions,  which,  when  evaluated 
against  reality,  provide  a  detailed  objective  assessment  of  SA.  This  type  of  assessment  provides 
a  direct  measure  of  SA.  We  describe  three  ways  of  gathering  this  information:  posttest,  during 
simulations,  or  during  interruptions  in  the  simulation. 

1 .  Posttest  questionnaires.  A  detailed  questionnaire  after  the  completion  of  each  simulated  trial, 
allowing  ample  time  to  respond.  Memories  of  dynamic  SA  will  be  less  reliable  with  time. 
People  tend  to  over-rationalize  and  over-generalize  about  past  mental  events  (Nisbett  & 
Wilson,  1977).  Kibbe  (1988)  used  this  technique  to  evaluate  SA  as  affected  by  automation 
of  a  threat  recognition  task. 

2.  Questionnaires  during  simulations.  One  way  of  overcoming  the  deficiencies  of  Post- 
Scenario  Questionnaires  is  to  ask  ATCSs  about  their  SA  while  they  are  carrying  out  their 
simulated  tasks.  It  is  possible  to  measure  response  time  (RT)  as  an  index  of  SA.  Durso  et  al. 
(1998)  recently  investigated  the  use  of  this  technique  in  an  ATC  task.  They  found  that  ATCS 
RT  to  probes  about  the  current  status  of  events  in  the  simulation  correlated  with  an  SME’s 
subjective  ratings  of  ATCS  performance.  Willems  and  Truitt  (1999)  found  an  increase  in  RT 
with  reduced  involvement;  however,  workload  increased. 

3.  Questionnaires  given  during  simulator  interruptions.  To  overcome  the  limitations  of 
reporting  on  SA  after  the  fact,  one  method  freezes  and  suspends  the  simulation  at  randomly 
selected  times,  while  ATCSs  answer  questions  about  the  current  situation.  Researchers 
compare  ATCS  perceptions  to  the  real  situation  as  an  objective  SA  measure.  Collecting  data 
in  this  manner  provides  an  objective  assessment  of  SA.  The  primary  disadvantage  of  this 
technique  is  the  temporary  halt  in  the  simulation.  Several  studies  have  used  this  technique  to 
collect  measures  of  SA  on  select  parameters.  Marshak,  Kuperman,  Ramsey,  and  Wilson 
(1987)  administered  queries  on  target  location,  altitude,  and  status  in  evaluating  various  map 
display  formats.  Fracker  (1989,  1990)  used  queries  to  measure  subject  knowledge  of  target 
identification  and  location  in  several  studies.  Mogford  and  Tansley  (1991)  used  queries 
regarding  aircraft  location  and  status  in  a  study  of  ATCSs.  One  potential  shortcoming  of 
obtaining  an  indication  of  SA  by  using  probes  on  a  few  predefined  elements  is  that  this  may 
have  an  effect  on  subjects’  attention  during  testing. 

Indirect  Measures  of  SA 

Researchers  may  expect  ATCSs  to  act  in  certain  ways  based  on  their  SA.  Some  information 
about  SA  may  be  determined  from  examining  behavior  on  specific  subtasks  of  interest.  Mosier 
and  Chidester  (1991)  found  that  high-performing  crews  had  fewer  verbal  communications  than 
poorer  performing  crews.  Rogers  (1994)  examined  answers  to  on-line  probes  from  the 
dispatcher  and  first  officer.  He  found  this  measure  sensitive  to  design  issues  surrounding 
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implementation  of  an  automated  system.  Other  behavioral  indices  might  include  time  to  make  a 
response  (verbal  or  non-verbal),  and  correct  or  incorrect  SA  as  identified  from  ATCS 
verbalizations  and  appropriateness  of  a  given  behavior  for  a  particular  situation. 

Performance  measures  provide  the  advantage  of  being  objective  and  are  usually  non-intrusive. 
Researchers  have  examined  specific  task  performance  as  an  indicator  of  SA.  Hansman  et  al. 
(1992)  used  detection  of  errors  as  a  measure  of  aircrew  SA.  They  found  that  it  was  sensitive  to 
differences  between  manual  and  automated  programming  modes  but  was  not  sensitive  to  the  use 
ofreadback.  Andre,  Wickens,  Moorman,  and  Boschelli  (1991)  measured  navigation 
performance  and  aircraft  control  and  found  it  was  not  sensitive  to  a  display  change  as 
anticipated. 

It  is  easy  for  subjects  to  bias  their  attention  to  the  single  issue  under  evaluation  in  a  particular 
study  through  predictability  and  response  priming.  For  example,  Busquets,  Parrish,  Williams, 
and  Nold  (1994)  measured  the  time  for  aircraft  to  respond  to  a  runway  incursion  as  a  measure  of 
pilot  SA.  This  is  a  good  example  of  a  situation  in  which  a  single  response  outcome  can  be 
expected  if  the  person  has  good  SA. 

One  of  the  challenges  in  measuring  SA  is  to  probe  ATCSs  without  them  needing  to  transform 
knowledge  into  a  format  that  we  can  measure.  Experts  are  notorious  for  their  inability  to 
verbalize  their  knowledge  or  present  a  solution  broken  down  in  small  steps.  They  will  often 
refer  to  a  “gut  feeling”  or  indicate  that  they  “just  know”  that  they  have  chosen  the  right  solution 
to  a  problem.  To  try  to  get  around  the  pitfalls  of  assessing  SA  in  expert  ATCSs,  we  have  created 
a  technique  that  will  leave  the  stimulus  (i.e.,  the  traffic  situation  displayed  on  the  radar  display) 
intact. 

Another  challenge  is  that  the  architecture  of  the  NAS  and  Host  prevents  temporarily  halting  the 
simulation  to  assess  SA.  We,  therefore,  needed  a  technique  that  allowed  us  to  probe  controllers 
for  SA  while  the  simulation  was  running.  Two  well-known  examples  of  techniques  that  use 
queries  during  simulations  and  ask  operators  about  situation-specific  information  while  they  are 
carrying  out  their  simulated  tasks  are  the  Situation  Presence  Assessment  Method  (SPAM)  and 
the  SAGAT. 

In  this  experiment,  we  assessed  SA  while  the  simulation  was  running  by  using  a  method  that 
incorporated  the  advantages  of  both  the  SPAM  and  the  SAGAT.  We  refer  to  this  method  as  the 
SAVANT.  We  also  incorporated  several  other  measures  of  SA  -  SAGAT,  SME  observer  ratings, 
and  self  reported  SA. 

3.1.1  SAVANT 


We  developed  an  objective  measure  based  on  responses  to  queries  that  incorporated  the 
advantages  of  both  the  SPAM  and  SAGAT  called  SAVANT.  We  used  this  technique  to  assess 
the  effects  of  changes  in  automation,  task  load,  and  position  on  ATCS  SA. 

In  SPAM,  one  of  the  experimenters  calls  the  ATCS  and  asks  a  question  about  two  aircraft.  The 
experimenter  then  measures  RT  to  queries  as  an  index  of  SA  (Durso  et  al.,  1998;  Willems  & 
Truitt,  1999).  The  implementation  of  SPAM  in  a  team  situation  is  difficult  because  one  would 
need  to  ask  both  R-  and  D-side  ATCSs  a  question  and  measure  the  RT.  Unless  more  than  one 
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experimenter  administers  the  SPAM  query,  the  manual  determination  of  the  RT  is  not  possible. 
Aside  from  the  difficulty  of  implementing  SPAM  in  an  ATCS  team  situation,  SPAM  has  the 
drawback  that  the  ATCS’s  task  during  the  SPAM  response  changes  to  a  visual  search  task.  An 
underlying  assumption  in  SPAM  is  that  the  ATCS  uses  the  radar  display  as  an  external  memory, 
and  the  time  to  find  the  information  is  an  indication  of  SA  (Durso  et  al.,  1 998).  One  way  to  use 
the  same  approach  as  SPAM  while  minimizing  the  search  task  is  to  use  the  radar  display  to  query 
the  ATCSs  by  highlighting  the  aircraft  involved  in  the  query.  This  has  the  advantage  that 
ATCSs  can  use  the  radar  display  to  respond  to  the  query,  eliminating  the  need  to  verbalize  the 
answer.  An  additional  advantage  is  that  we  can  time  the  start  of  the  query  and  the  start  and  end 
of  the  response  without  the  interference  of  the  experimenter. 

In  SAGAT,  the  experimenter  freezes  and  suspends  the  simulation  at  randomly  selected  times 
while  operators  answer  questions  about  the  current  situation.  The  primary  disadvantages  of  this 
technique  are  the  temporary  halt  in  the  simulation  and  the  use  of  a  separate  display  that  is 
different  from  the  radar  display.  Further,  one  potential  shortcoming  of  obtaining  an  indication  of 
SA  by  using  probes  on  a  few  predefined  elements  is  that  this  may  have  an  effect  on  participants’ 
attention  during  testing.  Queries  given  during  simulations  overcome  the  limitations  of  reporting 
on  situation-specific  information  after  the  fact. 

We  refer  to  the  queries  that  were  similar  to  SPAM  questions  as  Aircraft-pair  queries.  They 
assessed  more  relational  information  about  aircraft  pairs.  We  refer  to  the  queries  that  were 
similar  to  SAGAT  questions  as  Sector-Based  queries.  These  queries  assessed  more  global  SA. 

3 . 1 . 1 . 1  Data  Analysis  and  Results 

We  divided  the  SAVANT  data  into  several  data  sets  (Figure  1 1).  For  the  analyses,  we  separated 
the  Aircraft-Pair  queries  from  the  Sector-Based  queries.  We  then  conducted  analyses  on  a  subset 
of  the  Aircraft-Pair  data  containing  all  responses  and  a  subset  containing  correct  responses  only. 
These  subsets  contained  queries  about  current  and  future  situations.  For  the  subset  of  data 
containing  all  responses,  we  conducted  MANOVAs  on  RT  and  percent  correct  -  one  for  current 
and  one  for  future  queries.  For  the  subset  of  data  containing  correct  responses  only,  we 
conducted  two  ANOVAs  on  the  RT  -  one  for  current  and  one  for  future  queries.  For  the  Sector- 
Based  questions,  we  calculated  two  variables.  First,  we  calculated  the  percent  of  aircraft  ATCSs 
answered.  Secondly,  we  used  the  information  on  correct  answers  identified  by  the  SME  to 
calculate  the  percentage  of  correct  answers.  We  then  averaged  the  responses  for  the  four 
questions  that  asked  about  which  aircraft  were  not  in  communication,  did  not  conform  to 
clearances,  needed  a  hand  off  to  the  next  sector,  and  had  not  completed  clearances  yet.  In  all,  we 
had  five  data  sets.  The  MANOVA  for  the  Aircraft-Pair  questions  included  the  percentage  of 
responses  and  the  percentage  correctly  answered  relative  to  the  SME’s  answers.  We  conducted 
subsequent  ANOVAs  when  the  MANOVA  results  were  statistically  significant.  We  provide  the 
means,  SDs,  MANOVA,  and  ANOVA  results  in  Appendix  O. 
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Figure  1 1 .  Flow  chart  for  SAVANT  data  analysis. 

3  ■  1  ■  1  ■  1 . 1  Correct  Responses  Only  -  Aircraft-Pair  Questions 

We  conducted  two  separate  2x2x3  (position  x  task  load  x  automation)  within-subjects 
ANOVAs.  These  were  on  the  RT  for  correct  answers  only  for  the  current  and  future  SA 
questions,  respectively.  The  ANOVA  for  current  SA  questions  showed  a  significant  effect  for 
position  fF(l,15)  =  5.86 ,p<  .05,  Appendix  O,  Table  0-5].  When  sitting  on  the  R-side,  ATCSs 
responded  faster  to  the  current  SAVANT  questions  than  when  on  the  D-side  (Figure  12).  The 
ANOVA  for  future  SA  showed  a  significant  effect  for  task  load  [^(1,15)  =  10.81,  p  <  .01,  Table 
0-6],  ATCSs  responded  faster  to  the  future  SA  queries  when  task  load  was  low  (Figure  13). 
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Figure  12.  Response  time  for  current  correct  answers  by  position. 
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Figure  13.  Response  time  for  future  correct  answers  by  task  load. 

3 . 1 . 1 . 1 .2  All  Responses  Answered  -  Aircraft-Pair  Questions 

We  conducted  separate  2x2x3  (position  x  task  load  x  automation)  within-subjects  MANOVAs 
on  the  RT  and  percentage  correct  for  current  SA  and  future  SA  on  all  questions  answered, 
respectively.  The  multivariate  analysis  for  current  SA  showed  significant  results  for  position  and 
task  load  [A  =  .61,  F(2,14)  =  4.48 ,/?  <  .05;  A  =  .63,  F(2,14)  =  4.15,/?  <  .05,  respectively.  Table 
0-7].  The  multivariate  analysis  for  future  SA  showed  significant  results  for  task  load  [A  =  .62, 
F(2,14)  =  4.28,/?  <  .05,  Table  0-10].  Because  of  the  significant  multivariate  findings,  we 
conducted  subsequent  ANOVAs  and  used  an  adjusted  alpha  level  of  .025  to  determine 
significance. 

When  examining  ATCSs’  RTs  for  current  SA  queries,  we  did  not  find  significant  effects  for 
position  using  the  adjusted  alpha  level;  however,  a  trend  showed  that  R-side  ATCSs  responded 
faster  to  the  current  SA  queries  than  D-side  ATCSs  (Figure  14,  Table  0-8). 
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Figure  14.  Response  time  for  all  responses  of  current  SA  by  position. 


29 


In  addition,  a  secondary  trend  for  automation  showed  that  ATCSs  responded  faster  in  the  full 
automation  condition  than  in  the  limited  automation  condition.  There  were  no  differences 
between  full  and  no  automation  or  limited  and  no  automation  (Figure  15,  Table  0-8).  When 
examining  the  percentage  of  correct  responses,  the  data  did  not  reach  statistical  significance  nor 
was  a  trend  reached.  However,  when  task  load  was  high,  the  percentage  of  correct  responses 
was  higher  (Figure  16,  Table  0-9). 


No  Automation  Limited  Full  Automation 
Automation 


Figure  15.  Response  time  for  all  responses  of  current  SA  by  automation. 
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Figure  16.  Percent  correct  for  all  responses  of  current  SA  by  task  load. 

When  examining  ATCSs  RTs  for  future  SA  queries,  we  found  a  significant  effect  for  task  load 
[F(l,15)  =  7.57,/?  <  .05,  Table  0-11],  ATCSs  responded  faster  to  the  queries  when  task  load 
was  low  (Figure  17).  Analyses  for  the  percentage  of  correct  responses  did  not  show  any 
significant  effects. 
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Figure  17.  Response  time  for  all  responses  of  future  SA  by  task  load. 
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We  conducted  a  2  x  2  x  3  (position  x  task  load  x  automation)  within-subjects  MANOVA  for  the 
percentage  of  responses  and  the  percentage  of  correct  responses  relative  to  the  SME’s  answers. 
The  MANOVA  showed  a  significant  effect  for  position  [A  =  .44,  F(2,14)  =  8.70,  p  <  .05,  Table 
0-13].  Because  of  the  significant  multivariate  findings,  we  conducted  subsequent  ANOVAs  and 
used  an  adjusted  alpha  level  of  .025  to  determine  significance. 

The  univariate  analysis  for  the  percentage  of  correct  answers  showed  a  significant  effect  for 
position  [F(l,15)  =  13.40,p>  <  .01,  Table  0-15]  and  a  trend  for  automation  using  an  alpha  of  .05. 
When  on  the  R-side,  ATCSs  answered  a  significantly  higher  percentage  of  questions  correctly 
than  when  on  the  D-side  (Figure  18).  The  trend  for  automation  showed  that,  as  automation 
increased,  the  percentage  of  correct  responses  decreased  (Figure  19). 

The  univariate  analysis  for  the  percent  of  answers  did  not  show  any  significant  effects  for  the 
experimental  manipulations.  Analysis  of  the  six-participant  data  set  showed  a  significant  three- 
way  interaction  [A  =  .02,  F( 2,4)  =  21.83 ,p<  .05,  Table  0-16]  that  we  did  not  find  in  the  full  data 
set.  However,  the  univariate  follow-up  analyses  did  not  reach  statistical  significance.  We  did 
find  a  trend  for  the  position  x  automation  interaction  for  the  percentage  of  correct  responses. 
Automation  had  an  effect  for  R-side  ATCSs  but  not  for  D-side  ATCSs.  R-side  ATCSs’ 
percentage  of  correct  responses  was  higher  when  they  did  not  use  any  automation  (Figure  20, 
Table  0-18). 


Figure  18.  Percent  of  correct  responses  for  Sector-Based  questions  by  position. 
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Figure  19.  Percent  of  correct  responses  for  Sector-Based  questions  by  automation. 
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Figure  20.  Percent  of  correct  responses  for  Sector-Based  questions  by  position  and  automation 
(N=6). 

3. 1.1 .2  Discussion 

The  Aircraft-Pair  queries  assessed  SA  regarding  the  state  of  aircraft  in  the  airspace.  However, 
Sector-Based  queries  assessed  more  global  SA  for  action-oriented  items  such  as  whether  aircraft 
were  conforming  to  clearances.  This  contrast  in  queries  resulted  in  Aircraft-Pair  questions  being 
more  applicable  to  both  the  R-  and  D-side  ATCSs,  but  the  Sector-based  queries  were  more 
applicable  to  the  R-side  ATCSs.  The  Sector-Based  questions  focused  more  on  tasks  related  to 
the  tactical,  R-side  ATCS  (i.e.,  control  actions  communicated  to  aircraft).  The  R-side  ATCS 
needed  to  assess  the  overall  comprehension  of  the  actions  issued  to  and  performed  by  the 
aircraft,  and  the  D-side  ATCS  gathered  more  second-hand  information  about  the  state  of  the 
aircraft  location.  Because  of  these  differences,  we  expected  to  see  the  effect  of  experimental 
variables  for  both  the  R-  and  D-side  ATCSs  on  the  Aircraft-pair  queries.  Because  Sector-Based 
queries  focused  more  on  issues  related  to  tactical  control  of  aircraft,  we  expected  only  the  R-side 
to  be  affected  by  the  I  Vs. 

For  the  data  set  of  all  Aircraft-pair  responses,  a  trend  showed  better  SA  under  full  automation 
compared  to  limited  automation,  although  it  was  not  significantly  different  compared  to  no 
automation.  In  the  limited  automation  condition,  ATCSs  were  less  involved  with  the  traffic.  In 
contrast,  full  automation  provided  the  graphic  plan  display  that  is  a  representation  of  aircraft 
positions  and  conflict  information  visually,  allowing  the  D-side  ATCS  to  stay  involved.  Even 
under  no  automation,  the  ATCSs  had  better  SA  than  in  the  limited  automation.  In  the  field, 
ATCSs  training  includes  building  the  “picture”  from  flight  strips  and  maintaining  that  picture 
without  a  graphical  representation.  Therefore,  by  giving  them  flight  progress  strips,  they  may 
have  been  better  able  to  stay  involved  than  with  the  limited  automation  where  ATCSs  were  not 
familiar  with  the  format  of  displayed  aircraft  information. 

ATCSs  had  better  awareness  of  the  current  situation  when  working  on  the  R-side.  The  R-side 
ATCS  actively  controls  traffic,  and  the  D-side  ATCS  has  other  responsibilities  to  attend  to 
besides  focusing  on  the  radar  screen.  This  gives  R-side  ATCSs  an  advantage  because  they  are  in 
a  more  active  role  related  to  the  traffic  compared  to  the  D-side  who  is  in  more  of  a  monitoring 
role.  This  compliments  Willems  and  Truitt’s  (1999)  and  Endsley  and  Kiris’  (1995)  findings  that 
SA  was  lower  when  passively  processing  information  instead  of  actively  processing  it. 
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ATCSs  had  better  awareness  for  future  situations  when  task  load  was  low.  The  fewer  the 
aircraft,  the  easier  it  was  to  project  them  into  the  future,  increasing  SA.  Because  the  Aircraft- 
pair  questions  assessed  SA  relevant  to  both  positions,  task  load  did  not  affect  ATCS  SA  relative 
to  their  control  position. 

The  higher  percentage  of  correct  responses  in  the  high  task  load  condition  would  seem  to  be 
counter  to  what  we  would  expect.  We  believe  that  the  replacement  of  missing  data  affected  this 
result.  We  replaced  more  missing  data  in  the  high  task  load  condition  than  in  the  low  task  load 
condition.  However,  ATCSs  did  worse  than  what  guessing  would  account  for. 

Sector-based  questions  were  really  more  for  the  tactical,  R-side  ATCS.  When  analyzing  these 
questions,  we  focused  on  the  percentage  answered  and  the  percentage  correct.  We  did  not  use 
RT  as  an  indicator  of  SA  for  these  questions.  We  felt  that  RT  was  not  appropriate  because  the 
first  aircraft  chosen  would  have  a  high  RT,  and  subsequent  aircraft  tended  to  be  selected  in  quick 
succession.  Analysis  of  the  mean  for  the  percent  of  aircraft  answered  (the  number  of  responses 
given  by  ATCSs)  compared  to  the  mean  number  of  responses  chosen  by  the  SME  and  indicated 
that  ATCSs  did  have  enough  time  to  click  on  all  applicable  aircraft  to  the  chosen  query. 

SA  was  higher  for  these  action-oriented  items  when  ATCSs  were  on  the  R-side.  The  R-side 
needs  to  have  an  overall  comprehension  of  whether  aircraft  are  conforming  to  clearances;  in 
contrast,  the  D-side  ATCS  needs  to  know  the  state  of  the  aircraft.  In  addition,  a  trend  showed 
lower  SA  under  increasing  levels  of  automation.  This  coincides  to  findings  (Carmody  & 
Gluckman,  1993;  Endsley  &  Kiris,  1995)  that  SA  decreases  under  automation.  In  the  six 
participant  data  set,  level  of  automation  differentially  affected  SA  through  ATCS  position.  R- 
side  ATCSs’  SA  was  higher  when  no  automation  was  present.  These  results  indicate  that  the 
experimental  manipulations  affected  R-side  ATCSs  more.  When  no  automation  was  present,  the 
D-side  ATCS  could  devote  full  attention  to  the  DSR  screen  and  assist  the  R-side  ATCS,  which 
freed  up  some  cognitive  resources  of  the  R-side  ATCS.  When  no  automation  was  present,  the  D- 
side  ATCS  could  manipulate  the  data  blocks  allowing  for  easier  viewing  and  no  overlapping 
information. 

The  nature  of  the  motor  response  to  the  queries  resulted  in  slower  RTs  than  in  previous  studies 
(e.g.,  Endsley,  Sollenberger,  Nakata,  &  Stein,  2000;  Willems  &  Truitt,  1999).  Before  responding 
to  the  question,  the  ATCSs  had  to  grasp  the  mouse  and  move  it  into  position  to  answer  the  query, 
which  slowed  RT.  Therefore,  small  differences  may  be  more  significant  than  the  graphs  seem  to 
indicate.  Another  reason  that  may  make  it  difficult  to  compare  the  current  results  to  other  studies 
is  that  ATCSs  did  not  need  to  perform  a  search  task  because  the  SAVANT  screen  highlighted  the 
aircraft  in  question  for  easy  recognition. 

3.1.2  SAGAT 

Endsley  and  Kiris  (1995)  study  contains  full  documentation  on  SAGAT,  the  ATC  version. 
Normally,  SAGAT  measures  three  levels  of  SA  by  querying  the  ATCS  during  freezes  in  the  trial. 
In  the  current  study,  we  queried  participants  only  at  the  end  of  each  scenario.  We  terminated  the 
scenarios  between  33  and  40  minutes  to  prevent  participants  from  creating  a  memorization 
strategy  had  they  had  knowledge  about  the  time  that  SAGAT  questions  would  occur.  Each 
SAGAT  query  session  asked  ATCSs  about  the  items  in  Table  6  (Appendix  J). 
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Table  6.  Levels  of  SA  (from  Endsley) 


Level  1  SA  -  Perception  of  Traffic  Situation 

Levels  2  and  3  SA  —  Comprehension  and 
Projection  of  Traffic  Situation 

aircraft  position  (+/-  7  miles) 

aircraft  separation 

aircraft  callsign 

aircraft  next  fix 

aircraft  altitude  (+/-  300  feet) 

open  clearances 

aircraft  ground  speed  (+/-20  knots) 

clearance  reception 

aircraft  heading  (+/-  15  degrees) 

clearance  conformance 

aircraft  flight  path  change  (vertical,  turning) 

aircraft  communications 

aircraft  type 

flight  plan  conformance 

aircraft  level  of  control 

aircraft  flight  profile  (inbound,  en  route, 
outbound 

aircraft  hand-offs  needed 

Special  airspace  separation 

SAGAT  contained  questions  about  general  traffic,  specific  aircraft  inside  and  outside  active 
airspace,  future  status  of  specific  aircraft,  and  other  questions  designed  to  assess  SA.  We  divided 
the  SAGAT  queries  into  several  categories  based  on  the  level  of  SA  the  query  assessed  — 
perception,  comprehension,  or  projection.  Aircraft  position,  callsign  letter,  callsign  number, 
altitude,  speed,  and  heading  are  verbatim  information  from  taken  directly  off  the  DSR  display, 
and  we  categorized  them  as  addressing  Level  1  SA  -  Perception.  Aircraft  vertical  change  and 
turning  are  perceptual  information  gathered  from  the  display  over  time.  Aircraft  type  and  level 
of  control  are  also  categorized  as  basic  information  regarding  each  aircraft  that  we  categorized  as 
part  of  Level  1  S  A. 

Aircraft  separation  problems,  clearances  not  yet  completed  by  the  aircraft,  clearances  received 
correctly,  clearance  conformance,  aircraft  in  communication  with  ATCS,  handoffs  needed, 
projected  special  airspace  violations,  next  fix  (projection  of  future  flight  path),  and  flight  plan 
conformance  composed  the  Level  2/3  SA  category  (comprehension  of  the  status  of  the  aircraft 
relevant  to  ATCS  goals  and  projection  of  their  future  status). 

3. 1.2.1  Data  Analysis  and  Results 

We  compared  ATCSs’  perceptions  of  the  traffic  situation  as  reported  on  the  SAGAT  queries  to 
the  actual  state  of  the  traffic  situation  at  the  end  of  the  scenario.  We  scored  ATCSs’  answers  as 
correct  or  incorrect  (within  tolerance  levels  specified  in  Table  6)  and  subjected  them  to  an 
arcsine  transformation  to  correct  for  non-normality  of  binomial  data.  The  analyses  included 
MANOVAs  on  each  category  of  questions  and  subsequent  ANOVAs  when  the  MANOVA 
results  were  statistically  significant  and  to  assess  any  trends  in  the  data.  Appendix  P  provides  the 
means,  SDs,  MANOVA,  and  ANOVA  results.  The  graphs  depict  the  mean  from  the  transformed 
data  set  and  the  SD  from  the  raw  data  set. 

3.1.2. 1.1  Level  1  SA 

There  were  not  enough  degrees  of  freedom  to  perform  a  MANOVA  on  all  the  Level  1  SA  query 
items.  We  performed  2x2x3  (position  x  task  load  x  automation)  within-subjects  ANOVAs  on 
each  item  and  used  an  adjusted  alpha  level  of  .0056  to  test  for  significance.  We  found  several 
secondary  trends  among  the  Level  1  SA  items.  When  working  on  the  R-side  position,  ATCSs’ 
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awareness  of  aircraft  location  was  higher  than  when  on  the  D-side  (Figure  21,  Appendix  P,  Table 
P-4).  A  secondary  trend  of  task  load  for  callsign  letter  showed  that  ATCSs  identified  more 
callsign  letters  correctly  under  low  task  load  conditions  (Figure  22,  Table  P-5). 
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Figure  21 .  Percent  correct  for  aircraft  position  by  position. 
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Figure  22.  Percent  correct  for  awareness  of  aircraft  callsign  letter  by  task  load. 

For  headings,  we  found  a  secondary  trend  of  the  task  load  x  automation  interaction.  Under  full 
automation  and  high  task  load,  ATCSs’  awareness  of  headings  was  higher  than  full  automation 
and  low  task  load  (Figure  23,  Table  P-9).  Awareness  for  aircraft  type  was  highest  in  the  high 
task  load,  limited  automation  condition  (Figure  24,  Table  P-12). 


No  Automation  Limited  Automation  Full  Automation 
»  -  Low  Task  Load  — High  T ask  Load 


Figure  23.  Percent  correct  for  awareness  for  headings  by  task  load  and  automation. 
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Figure  24.  Percent  correct  for  awareness  of  aircraft  type  by  task  load  and  automation. 

3.1 .2. 1.2  Level  2  SA 

There  were  not  enough  degrees  of  freedom  to  perform  a  MANOVA  on  all  the  Level  2  SA  query 
items.  We  performed  2x2x3  (position  x  task  load  x  automation)  within-subjects  ANOVAs  on 
each  item  and  used  an  adjusted  alpha  level  of  .006  to  test  for  significance.  Secondary  trends  in 
the  data  indicated  that  awareness  of  the  next  fix  was  lowest  under  limited  automation  (Figure  25, 
Table  P-14).  When  on  the  R-side,  ATCSs’  awareness  of  aircraft  flight  profiles  was  higher  than 
when  on  the  D-side  (Figure  26,  Table  P-15).  A  secondary  trend  of  task  load  for  awareness  of 
separation  showed  that  ATCSs  had  a  higher  awareness  of  separation  issues  under  low  task  load 
conditions  (Figure  27,  Table  P-16).  Under  high  task  load  conditions,  ATCSs  were  more  aware 
of  aircraft  clearances  received  (secondary  trend;  Figure  28,  Table  P-18).  Awareness  for 
clearance  conformance  was  lowest  under  the  low  task  load,  no  automation  condition  (secondary 
trend;  Figure  29,  Table  P-19). 


No  Automation  Limited  Automation  Full  Automation 


Figure  25.  Percent  Correct  for  awareness  of  next  fix  by  automation. 
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D-side  R-side 

Figure  26.  Percent  Correct  for  awareness  of  flight  profile  by  position. 


Low  Task  Load  High  Task  Load 

Figure  27.  Percent  correct  for  awareness  of  aircraft  separation  by  task  load. 


Low  Task  Load  High  Task  Load 

Figure  28.  Percent  correct  for  awareness  of  aircraft  clearances  received  by  task  load. 


37 


Automation 

-  ♦  Low  Task  Load  . i  High  Task  Load 

Figure  29.  Percent  correct  for  awareness  of  aircraft  conformance  by  task  load  and  automation. 
3. 1.2. 1.3  Level  3  SA 

We  found  no  significant  effects  or  trends  for  the  variables  assessing  Level  3  SA. 

3.1.2.2  Discussion 


Differences  in  automation  had  an  impact  on  ATCSs’  awareness  for  the  aircrafts’  next  fix. 
Awareness  for  the  aircraft’s  next  fix  was  lowest  under  limited  automation  conditions.  In  contrast 
to  full  automation  or  no  automation,  the  limited  automation  did  not  allow  ATCSs  to  maintain  the 
“picture”  as  well.  In  limited  automation,  ATCSs  only  had  electronic  flight  strips  and  could  not 
pull  up  the  graphical  plans  display  resembling  the  DSR  display.  This  appeared  to  hamper  their 
ability  to  stay  up  with  next  fix  information.  ATCSs  have  traditionally  trained  to  use  paper  flight 
strips  to  help  build  their  picture,  and,  in  the  no  automation  condition,  they  may  have  used  this  to 
stay  involved.  Similarly,  they  may  have  used  the  graphical  display  provided  in  the  full 
automation  to  maintain  the  picture. 

Even  though  ATCS  scores  for  aircraft  positions  were  low,  we  did  find  that  R-side  ATCSs  had 
better  awareness  of  aircraft  location  than  D-side  ATCSs.  R-side  ATCSs  spend  more  time 
viewing  the  radar  display  and  actively  controlling  traffic,  and  the  D-side  ATCSs  assist  and  are  in 
a  more  monitoring  role.  The  active  control  of  traffic  may  increase  SA  as  reflected  in  this  item 
and  supported  by  Willems  and  Truitt’s  (1999)  findings  that  SA  was  higher  for  active  ATCSs 
than  monitoring  ATCSs.  We  had  a  similar  finding  for  awareness  of  flight  profile.  The  R-side 
ATCSs  were  better  aware  of  aircraft  flight  profiles  -  en  route,  outbound  from  an  airport,  or 
inbound  to  an  airport  in  the  sector  -  than  the  D-side  ATCSs.  This  corresponds  well  with  the 
notion  that  one  has  better  memory  for  things  that  you  do  yourself  (the  R-side  radioed  the  aircraft 
and  discussed  changes  in  flight  profile  with  the  pilots)  than  things  that  are  done  for  you  (the 
D-side  received  second-hand  information  after  the  R-side  initiated  the  control  action)  (Slamecka 
&  Graf,  1978,  as  cited  in  Albright,  Truitt,  Barile,  Vortac,  &  Manning,  1994). 

Under  high  task  loads,  ATCSs  were  less  aware  of  aircraft  callsign  and  aircraft  separation; 
however,  ATCSs’  awareness  for  receipt  of  clearances  was  higher  than  under  low  task  load 
conditions.  In  SAG  AT,  we  queried  ATCSs  only  on  those  aircraft  that  they  initially  placed  on  the 
SAGAT  map.  Under  low  task  load,  ATCSs  may  be  able  to  recall  more  aircraft  than  the  ones  that 
required  their  attention.  Under  high  task  load,  ATCSs  most  likely  had  so  many  aircraft  that 
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required  their  attention,  that  they  could  not  recall  all  of  them.  Therefore,  when  queried  about 
whether  an  aircraft  that  they  recalled  had  received  a  clearance,  under  low  task  load,  they  may 
have  remembered  less  information  correctly  because  the  recalled  aircraft  included  unimportant 
aircraft  as  well.  Under  high  task  load,  all  recalled  aircraft  may  have  been  important  and  whether 
or  not  one  of  these  aircraft  had  received  a  clearance  now  was  important  to  the  ATCS,  resulting  in 
higher  scores. 

Task  load  also  had  an  impact  on  several  other  variables;  however,  the  level  of  automation  altered 
that  effect.  For  awareness  of  aircraft  headings,  task  load  had  an  impact  for  this  item  under  full 
automation  conditions;  ATCSs  were  more  aware  of  headings  when  task  load  was  high.  With  full 
automation,  ATCSs  had  conflict  indications  and  reached  out  more  to  resolve  those  conflicts, 
bringing  the  headings  for  those  aircraft  to  their  attention.  In  addition,  these  aircraft  may  have 
been  important  because  all  recalled  aircraft  under  high  task  load  had  the  attention  of  ATCSs. 
ATCSs  had  higher  SA  for  aircraft  type  under  limited  automation  and  high  task  load  conditions. 

In  limited  automation,  ATCSs  could  easily  access  aircraft  type  from  the  aircraft  list.  Under  high 
task  load,  this  information  seems  to  have  more  importance  to  the  ATCSs.  Awareness  of  aircraft 
conformance  to  clearances  was  lowest  under  low  task  load,  no  automation.  Under  low  task  load, 
ATCSs  may  have  remembered  less  information  correctly  because  the  recalled  aircraft  included 
unimportant  aircraft  as  well  important  aircraft.  Further,  they  did  not  have  access  to  conflict 
indications  and  were  not  as  concerned  with  possible  conflicts  because  the  likelihood  of  them  was 
low.  In  contrast,  under  high  task  load,  recalled  aircraft  included  only  important  aircraft  and  the 
potential  for  possible  conflicts  was  large,  therefore,  even  under  no  automation  conditions, 

ATCSs  paid  great  attention  to  aircraft  conformance. 

Although  Endsley  (2001)  validated  aircraft  placement  from  a  larger  screen  to  a  smaller  screen  for 
SAGAT  with  pilots  and  found  a  5.21  mile  error  within  the  tolerance  band  used  in  the  current 
study,  the  overall  percentage  of  correct  responses  for  aircraft  positions  was  quite  low, 
approximately  20%.  There  are  several  explanations  for  the  low  percentage  associated  with 
awareness  for  aircraft  position.  First,  the  discrepancy  in  size  between  the  DSR  screen  and  the 
computer  monitor  may  physically  create  a  scaling  problem,  making  it  difficult  for  ATCSs  to 
accurately  place  aircraft  at  positions  coinciding  with  the  DSR  position.  The  large  display  used  in 
Endsley’ s  study  was  approximately  19  in.;  in  the  current  study,  the  DSR  display  was  29  in.  On 
the  other  hand,  the  change  in  scale  between  the  29-in.  DSR  display  and  the  SAGAT-probe 
display  may  have  caused  an  additional  cognitive  challenge  for  ATCSs.  We  have  not  established 
that  a  change  in  scale  does  not  distort  ATCS  recall  for  aircraft  positions.  The  probe  itself  also 
may  not  access  the  relevant  information  within  the  ATCSs’  knowledge.  ATCSs  may  have  the 
knowledge  for  aircraft  positions  relative  to  one  another  but  cannot  express  them  at  the  exact 
location  as  required  by  the  format  provided  by  the  probe.  Or,  perhaps,  the  ATCSs  really  do  not 
have  the  information  stored  in  memory  but  instead  use  the  radar  display  as  an  external  memory. 
If  this  is  the  case,  then  a  good  indication  of  SA  may  be  the  speed  at  which  they  can  find  the 
information  from  the  external  memory.  For  instance,  awareness  of  aircraft’s  next  fix  and  aircraft 
type,  although  not  available  directly  from  the  radar  display,  can  be  accessed  from  flight  plan 
information.  As  pointed  out  by  Durso  et  al.  (1998),  even  though  information  is  not  in  memory, 
knowing  where  to  find  it  can  imply  good  SA.  Helbing  (1997)  suggested  that  ATCSs  use  aircraft 
positions  as  memory  pegs  (i.e.,  they  store  relevant  information  about  aircraft  by  aircraft  position 
and  are  not  necessarily  able  to  retrieve  aircraft  information  when  probed  using  cues  such  as 
callsign).  Helbing’s  argument  suggested  that  ATCSs  store  aircraft  information  in  memory  using 
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positional  memory  pegs.  The  current  research  findings  suggest  that  ATCSs  do  not  have  good 
memory  for  exact  aircraft  locations.  Therefore,  if  Helbing’s  model  is  accurate,  the  spatial 
representation  that  ATCSs  use  to  store  aircraft  information  is  not  simply  a  scaled  version  of  the 
actual  displayed  information. 

Albright  and  Lewandowsky  (1995)  provide  an  additional  explanation  for  the  low  percent  correct 
on  recall  of  aircraft  positions.  These  researchers  showed  that  in  a  low  complexity  ATC  task, 
subjects  displayed  implicit  momentum  (i.e.,  when  probed,  subjects  thought  aircraft  locations 
were  slightly  ahead  of  their  actual  location).  Although  tolerance  bands  used  in  SAGAT  may 
have  absorbed  this  effect,  in  this  experiment,  considering  the  level  of  traffic  task  load  (necessary 
to  justify  the  use  of  an  R-  and  a  D-side),  such  a  forward  projection  may  cause  the  low  level  of 
correct  recall  of  aircraft  positions. 

Results  for  other  Level  1  SA  verbatim  items  showed  that  ATCSs’  awareness  for  aircraft  callsign 
was  less  than  20%,  even  lower  than  for  aircraft  position.  In  contrast,  ATCSs’  awareness  for 
aircraft  headings  reached  60%.  Implications  point  toward  lower  recall  for  alphanumeric 
information  and  higher  for  graphical  information.  Because  ATCSs  can  derive  awareness  of 
aircraft  heading  by  formatting  the  orientation  of  the  vector  lines,  this  gives  them  an  advantage  in 
recall.  The  results  clearly  show  we  cannot  expect  ATCSs  to  recall  verbatim  information.  This 
does  not  mean  that  they  do  not  have  that  information  stored  in  memory;  it  simply  means  that  the 
information  is  not  accessible  to  us  for  probing.  Therefore,  in  future  studies,  querying  ATCSs  for 
verbatim  information  will  have  no  added  value. 

Awareness  of  receipt  of  aircraft  clearances  and  conformance  to  clearances.  Level  2  SA,  was 
high.  Although,  in  general,  awareness  for  Level  1  SA  items  was  relatively  low,  there  did  not 
seem  to  be  an  adverse  effect  for  Level  2  SA.  One  could  use  the  SAGAT  results  to  infer  that  the 
information-processing  model  proposed  by  Endsley  and  Smolensky  (1998)  is  not  correct  (i.e.,  an 
ATCS  would  not  need  Level  1  SA  before  achieving  Level  2  SA).  However,  the  results  from  this 
experiment  do  not  indicate  whether  ATCSs  have  processed  information  from  Level  1  SA  into 
other  variables  or  formats  or  that  the  raw  information  is  still  present  but  cannot  be  directly 
measured  by  Level  1  SA  queries. 

We  did  not  have  significant  findings  for  Level  3  SA,  the  projection  of  elements  into  future 
situations.  This  clearly  indicates  that  even  with  the  DST  that  should  have  moved  ATCSs  to  a 
strategic  orientation,  our  ATCSs  still  controlled  traffic  in  a  mostly  tactical  fashion.  This,  of 
course,  could  also  be  because  our  ATCSs  were  novice  DST  users.  It  would  be  interesting  to  test 
ATCSs  that  have  extensive  experience  using  DST  and  compare  those  SA  results  with  our  current 
findings.  One  would  hope  to  find  that  for  the  experienced  DST  users,  level  3  SA  has  increased 
compared  to  our  novice  DST  users. 

Direct  comparison  across  studies  is  difficult  due  to  different  traffic  task  loads  and  different 
numbers  of  ATCSs  working  a  sector.  In  the  present  study,  much  higher  traffic  task  loads  were 
present  (mean  of  25  in  low  task  load  and  mean  of  35  in  high  task  load)  than  in  previous  studies 
(e.g.,  mean  of  19  in  low  traffic  task  load  and  24  in  high  traffic  task  load  scenarios)  (unpublished 
data  extracted  from  original  data  from  Willems  &  Truitt,  1999).  However,  in  the  current  study, 
we  used  both  an  R-side  and  D-side  ATCS  and  previous  studies  used  only  an  R-side  ATCS. 
Previous  research  shows  that  there  is  a  marked  effect  of  traffic  task  load  on  SA  as  the  number  of 
aircraft  present  exceeds  approximately  12  to  15  (Endsley  &  Rodgers,  1998).  ATCSs  will  attend 
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to  only  a  subset  of  more  important  aircraft,  thus  lowering  the  percentage  of  aircraft  attended  to 
overall.  The  much  lower  absolute  number  correct  in  the  present  study  (mean  of  4. 1 1  per  trial), 
however,  indicates  far  more  than  a  workload  effect.  It  is  most  likely  reflecting  a  lowering  of 
awareness  for  aircraft  that  had  ceased  to  be  important  after  the  ATCS  handed  them  off.  Another 
explanation  for  this  may  be  that  with  the  time  pressure  of  the  current  traffic,  ATCSs’  dynamic 
memory  span  reverted  to  approximately  three  items  (Moray,  1986).  Under  lower  traffic  task 
loads  and  with  only  an  R-side  ATCS,  Willems  and  Truitt  found  recall  for  10  items.  Therefore, 
the  results  presented  here  reflect  lower  SA  than  would  be  present  for  active  traffic  and  perhaps 
the  pattern  of  attention  across  elements  of  the  situation  would  be  different. 

3.1.3  Post-Scenario  Questionnaire 

3 . 1 .3 . 1  Data  Analysis  and  Results 

We  conducted  a  2  x  2  x  3  (position  x  task  load  x  automation)  within-subjects  MANOVA  on 
three  items  assessing  participants’  self-reported  SA.  These  items  included  SA  for  current 
aircraft,  projected  aircraft  locations,  and  potential  violations.  The  multivariate  analysis  showed 
significant  results  for  position  and  task  load  [A  =  .52,  F(3,13)  =  3.95,  p  <  .05  and  A  =  .32, 
F(3,13)  =  9.30,  p  <  .001],  respectively.  Appendix  Q,  Table  Q-2.  Because  the  effects  of  position 
and  task  load  were  significant  at  the  multivariate  level,  we  conducted  subsequent  ANOYAs  for 
each  SA  item  and  used  an  adjusted  alpha  of  .017  to  determine  significance.  See  Appendix  Q  for 
the  means,  SDs,  MANOVA,  and  ANOVA  tables. 

When  examining  ATCSs’  SA  for  current  aircraft  locations,  we  found  significant  effects  for  both 
position  and  task  load  [F(l,15)  =  11.33,  p  <.  01  and  F(l,  15)  =  11.91,/?  <  .01],  respectively. 
Figure  30,  Table  Q-3.  When  in  the  R-side  position,  ATCSs  rated  their  perceived  SA  for  current 
aircraft  locations  higher  than  when  they  were  in  the  D-side  position.  Further,  when  controlling 
traffic  in  a  low  task  load  scenario,  ATCSs  perceived  they  had  higher  SA  for  current  aircraft 
locations  than  in  high  traffic  task  load  scenarios. 


Figure  30.  SA  for  current  aircraft  locations  by  task  load. 

For  the  item  assessing  SA  for  projected  aircraft  locations,  only  a  significant  effect  of  task  load 
occurred  [F(l,15)  =  12.76, p  <  .01,  Figure  31,  Table  Q-4],  Once  again,  ATCSs  perceived  they 
had  higher  SA  for  projected  aircraft  locations  when  controlling  traffic  in  a  low  task  load  scenario 
compared  to  a  high  task  load  scenario. 
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Low  Task  Load  High  Task  Load 


Figure  3 1 .  SA  for  projected  aircraft  locations  by  task  load. 

When  asked  to  rate  their  SA  for  potential  violations,  ATCSs  reported  higher  levels  of  SA  under 
low  task  load  scenarios  in  comparison  to  high  task  load  scenarios  [F(l,15)  =  22.21, p  <  .001, 
Table  Q-5].  The  univariate  results  showed  a  secondary  trend  for  the  interaction  between  task 
load  and  automation  (Table  Q-5).  As  shown  in  Figure  32,  the  secondary  trend  indicated  task 
load  had  the  largest  effect  under  the  full  automation  condition.  When  ATCSs  used  full 
automation,  their  SA  for  potential  violations  was  lower  when  the  task  load  was  high  compared  to 
low. 
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Figure  32.  SA  for  potential  violations  by  task  load  and  automation. 
3. 1.3.2  Discussion 


When  assessing  SA  for  potential  violations,  the  degree  of  automation  affected  the  impact  of  task 
load.  The  trend  indicated  that  when  automation  was  not  present  or  was  limited,  the  effects  of 
high  and  low  task  load  were  not  significantly  different.  In  contrast,  when  ATCSs  used  full 
automation,  high  task  loads  led  to  ratings  of  lower  SA  for  potential  violations.  This  finding 
relates  to  other  findings  that  indicated  ATCSs  felt  the  DST  behavior  was  less  predictable  when 
task  load  was  high  and  full  automation  was  present.  In  the  case  of  the  DST,  the  ATCS  had  to 
constantly  monitor  it  to  fully  utilize  it.  Constant  monitoring  of  automation  induces  higher 
workload  and  can  result  in  lower  levels  of  SA.  Under  low  task  load  conditions,  the  ATCS  may 
be  able  to  compensate  for  this,  but,  when  task  load  is  high,  the  ATCS  is  less  able  to  maintain  the 
picture  and  maintain  SA.  The  ATCS’s  experience  level  using  the  automation  may  have 
contributed  to  lower  SA  for  potential  violations  under  high  task  loads.  Our  participants  were 
novice  DST  users,  and,  as  such,  their  skill  level  using  the  DST  was  not  very  high.  Under  low 
task  load  conditions,  they  had  enough  time  to  compensate  for  their  lack  of  experience  with  the 
tool;  however,  under  high  task  load  conditions,  their  lower  skill  level  with  the  DST  became  a 
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factor  influencing  SA.  We  would  expect  that  with  additional  training  and  experience  using  the 
DST,  we  would  not  see  such  a  drastic  drop  in  SA  under  high  task  load  conditions.  The  amount 
of  training  and  experience  using  the  DST  to  offset  this  detriment  in  SA  would  need  to  be 
investigated. 

Task  load  had  a  significant  impact  on  all  SA  items.  Increasing  task  load  led  to  decreases  in 
perceived  SA  for  current  aircraft  locations,  projected  aircraft  locations,  and  potential  violations. 
Results  imply  that  higher  task  load  scenarios  had  an  impact  on  the  cognitive  resources  ATCSs 
allocated  to  SA.  When  task  load  was  low,  ATCSs  were  able  to  keep  focus  on  the  big  picture  and 
displayed  higher  SA. 

Position  affected  SA  for  current  aircraft  locations.  R-side  ATCSs  rated  their  SA  higher  along 
this  dimension  than  D-side  ATCSs.  This  item  reflects  the  differences  in  the  responsibilities  of 
the  R-  and  D-side  ATCSs.  The  R-side  ATCS  actively  scanned  the  DSR  screen  and  directed 
traffic,  but  the  D-side  ATCS  had  other  responsibilities  that  did  not  focus  on  the  scope  leading  to 
lower  SA. 


3.1.4  Over-the-  Shoulder 
3 . 1 .4. 1  Data  Analysis  and  Results 

Maintaining  awareness  of  aircraft  positions,  ensuring  positive  control,  detecting  pilot  deviations 
from  control  instructions,  and  correcting  own  errors  in  a  timely  manner  composed  the 
maintaining  attention  and  SA  category.  Appendix  R  contains  the  means,  SDs,  MANOVA,  and 
ANOVA  tables. 

The  MANOVA  showed  a  significant  effect  of  task  load  across  the  set  of  DVs  [A  =  .07,  F(4,4)  = 
12.52,  p  <  .05,  Table  R-2],  The  SME  felt  that  ATCSs  maintained  awareness  of  aircraft  positions, 
ensured  positive  control,  and  corrected  their  own  errors  in  a  timely  manner  better  when  the  task 
load  was  low  [F(l,7)  =  40.69, p  <  .001,  F(l,7)  =  56.52 ,p  <  .001,  and  F(l,7)  =  20.05,/?  <  .01, 
respectively,  Figure  33,  Appendix  R,  Table  R-3,  Table  R-4,  and  Table  R-6].  A  trend  in  the  data 
suggests  the  ATCSs  also  detected  pilot  deviations  from  control  instructions  better  under  low  task 
load  conditions. 
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Figure  33.  Maintaining  attention  and  SA  by  task  load. 
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3. 1.4.2  Discussion 


Interestingly,  automation  did  not  have  an  effect  on  the  SME’s  ratings  for  SA.  However,  we 
expected  that  automation  would  have  an  effect  on  SA  items  from  the  OTS.  These  results  imply 
that  the  change  in  automation  levels  did  not  affect  ATCSs  in  any  discemable  manner  along  the 
maintaining  attention  and  SA  items,  as  observed  by  the  SME.  Due  to  limitations  in  resources, 
we  used  only  one  SME  to  complete  OTS  forms.  The  SME  predominantly  focused  on  the  R-side 
ATCS.  We  wrongly  assumed  that  any  changes  in  the  team  should  be  shown  on  both  sides.  The 
R-side  functioned  as  if  everything  was  normal,  but  the  D-side  was  using  the  automation.  The 
automation  did  not  directly  affect  the  R-side  as  it  did  the  D-side,  as  viewed  by  the  SME. 
Therefore,  an  automation  effect  was  not  found. 

Task  load  affected  the  SME’s  ratings  of  ATCSs’  SA.  High  task  loads  contributed  to  lower  SA 
ratings.  Once  again,  high  task  loads  made  it  difficult  for  the  ATCS  to  maintain  the  big  picture. 
This  led  to  ATCSs  loosing  SA. 

3.2  Workload 


In  this  study,  we  used  ATWIT,  NASA  TLX,  and  PSQs  to  assess  participant  workload. 

3.2.1  Air  Traffic  Workload  Input  Technique 

Stein  (1985)  first  introduced  ATWIT,  which  is  an  online  measure  that  requires  ATCSs  to 
indicate,  at  set  times,  their  perception  of  their  current  workload.  ATWIT  is  an  instantaneous 
probe  that  investigates  overall  perceived  workload.  Contrary  to  the  NASA  TLX  (Hart  & 
Staveland,  1988),  the  participants  do  not  need  to  break  down  their  workload  by  origin.  Another 
advantage  of  the  ATWIT  over  post-scenario  ratings  of  workload  is  that  ATWIT  measures 
workload  during  the  simulation  instead  of  relying  on  participant’s  memory  after  the  scenario. 

The  ATWIT  measure  is  a  workload  estimate  based  on  a  scale  from  1  to  10  (Appendix  F  contains 
the  detailed  instructions  that  accompany  the  ATWIT  device).  The  anchors  used  for  the  ATWIT 
scale  relate  directly  to  the  task.  The  ATWIT  device  is  a  touch  sensitive  panel  that  displays  a  start 
button  at  the  beginning  of  the  simulation  and  a  10-point  scale  during  the  simulation.  The 
ATCSs,  prompted  by  a  low  tone,  made  a  workload  rating  every  3  minutes.  Each  participant 
made  10  ATWIT  ratings  in  a  scenario  allowing  calculation  of  the  mean  and  maximum  rating  for 
each  scenario. 

3.2.1 .1  Data  Analysis  and  Results 

We  created  four  workload-related  data  sets.  The  first  two  data  sets  contained  12x16  (scenarios 
x  ATCSs)  records  that  included  the  summary  variables  calculated  per  scenario  for  the  full  data 
set  and  the  six-participant  data  set.  The  other  data  sets  contained  12  x  10  x  16  (scenarios  x 
intervals  x  ATCSs)  records  containing  the  summary  variables  calculated  per  3-minute  interval 
for  the  full  and  partial  data  sets. 

To  analyze  the  effect  of  the  IVs  on  the  subjective  ratings,  we  used  a  MANOVA  on  the  mean 
ATWIT  ratings  and  mean  response  times  (RTs).  This  MANOVA,  structured  as  a  2  x  2  x  3 
(position  x  task  load  x  automation)  repeated  measures  design,  addressed  the  differences  across 
scenarios.  We  also  conducted  a  2  x  2  x  3  (position  x  task  load  x  automation)  ANOVA  on  the 
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maximum  ATWIT  ratings.  To  investigate  the  effect  of  time-on-task,  we  used  ANOVAs  on 
ATWIT  ratings  and  RTs  ina2x2x3xl0  (position  x  task  load  x  automation  x  interval) 
repeated  measures  design  because  we  did  not  have  enough  degrees  of  freedom  for  a  MANOV A. 
See  the  tables  in  Appendix  S  for  detailed  results.  We  do  not  present  redundant  results  from  the 
scenario  and  interval  data  sets. 

3 .2. 1 . 1 . 1  Mean  ATWIT  Ratings  and  Response  Times 

The  2  x  2  x  3  (position  x  task  load  x  automation)  within-subjects  MANOVA  showed  significant 
effects  for  position,  task  load,  automation  and  the  task  load  x  automation  interaction  [A  =  .36, 
F(2,14)  =  12.49,  p  <  .01,  A  =  .11,  F(2,14)  =  54.82, p  <  .001,  A  =  .43,  F(4,12)  =  3.92 ,p  <  .05,  A  = 
.62,  F(2,14)  =  4.38,7?  <  .05,  respectively,  Appendix  S,  Table  S-3].  Because  of  the  significant 
MANOVA  results,  we  performed  subsequent  ANOVAs.  We  used  an  adjusted  alpha  of  .025. 

The  ANOVA  for  the  mean  ATWIT  rating  showed  a  significant  effect  for  task  load  [F(l,15)  = 
110.20,7?  <  .0001,  Table  S-4].  ATCSs’  perceived  workload  increased  with  task  load. 
Automation  did  not  significantly  affect  perceived  workload.  The  analysis  did  reveal  a  secondary 
trend  that  showed  that  perceived  workload  increased  with  increasing  automation  (Figure  34). 


No  Automation  Limited  Automation 

tIt—  Full  Automation 


Figure  34.  Mean  ATWIT  ratings  by  task  load  and  automation. 

The  ANOVA  examining  RTs  showed  a  significant  effect  for  position  [F(l,15)  =  24.81,7?  <  .001, 
Table  S-5]  and  a  significant  interaction  between  position  and  task  load  [F(l,15)  =  7.03,  p  <  .05, 
Table  S-5].  ATCSs  responded  to  the  ATWIT  device  faster  when  working  on  the  R-side  than  the 
D-side.  The  level  of  task  load  affected  this.  The  simple  effect  of  position  within  low  task  load 
was  significant  [F(l,15)  =  30.94,7?  <  .001,  Table  S-5].  ATCSs  responded  faster  when  sitting  on 
the  R-side  compared  to  the  D-side;  this  effect  was  not  significant  when  task  load  was  high 
(Figure  35). 
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D  side  — ■ —  R  side 


Figure  35.  Mean  AT  WIT  response  time  by  task  load  and  position. 

3 .2 . 1 . 1 .2  Maximum  AT  WIT  Ratings 

The  ANOVA  examining  the  maximum  ATWIT  ratings  showed  significant  effects  for  task  load 
and  automation  [F(l,15)  =  145.18,/?  <  .0001  and  F(2,30)  =  3.44 ,p  <  .05,  respectively,  Figure  36, 
Table  S-9].  Increasing  task  load  caused  an  increase  in  maximum  ATWIT  ratings.  When  in  the 
full  automation  condition,  ATCSs  rated  their  maximum  workload  higher  than  when  they  were  in 
the  limited  automation  condition.  The  full  and  no  automation  and  limited  and  no  automation 
conditions  were  not  statistically  different. 


Low  Task  Load  High  Task  Load 


Figure  36.  Maximum  ATWIT  rating  by  task  load  and  automation. 

3.2.1 .1,3  Effect  of  Time-On-Task 

We  conducted  separate  2x2x3x10  (position  x  task  load  x  automation  x  interval)  ANOVAs  on 
ATCSs’  ratings  and  RTs  because  there  were  not  enough  degrees  of  freedom  to  perform  a 
MANOVA.  We  analyzed  the  effect  of  time-on-task  using  two  separate  ANOVAs  and  adjusted 
the  alpha  level  to  .025.  We  present  results  that  are  unique  to  only  this  data  set  and  are  not 
redundant  with  previously  reported  findings. 

Examination  of  the  simple  effects  of  task  load  within  interval  showed  significant  results  for  task 
load  within  each  interval  [Fs(l,15)  =  54.30,  57.68,  88.02,  72.76,  85.71,  130.94,  117.23,  94.85, 
96.75,  130.04  ,  all/?s  <  .001,  respectively,  Figure  37,  Table  S-l  1].  Over  time,  workload  under 
low  task  load  slowly  decreased,  but  under  high  task  load,  it  remained  relatively  constant.  The 
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Figure  37.  Mean  ATWIT  ratings  by  task  load  and  interval. 

results  from  the  ANOVA  using  the  last  six  participants  showed  slightly  different  results.  We 
found  significant  position  x  task  load  interaction  [.F(l,5)  =  15.61,/?  <  .05,  Figure  38,  Table  S-13]. 
The  simple  effects  of  task  load  were  significant  for  both  positions  [Fs(l,5)  =  105.71  and  3 1.00, 
ps  <  .01,  respectively,  Table  S-13].  The  simple  effect  of  position  within  high  task  load  was 
significant  [F(l,5)  =  6.90  ,p  <  .05,  Table  S-13],  but  the  simple  effect  of  position  within  low  task 
load  was  not  significant  [7^(1, 5)  =  .62,  p  >  .05,  Table  S-13].  The  level  of  task  load  affected 
ATCSs’  perceived  workload  when  they  were  in  both  positions,  although  this  effect  was  not  as 
strong  when  they  were  in  the  D-side  position  (Figure  38). 


+  .  Low  Task  Load  ~~m~~  High  Task  Load 

Figure  38.  Mean  ATWIT  ratings  by  task  load  and  position  (N=6). 

3.2.1 .2  Discussion 

Full  automation  did  not  statistically  differ  from  the  no  automation  conditions,  although  a  trend 
did  indicate  an  increase  in  perceived  workload.  According  to  Bressolle,  Benhacene,  Boudes,  and 
Pari  (2000),  the  introduction  of  automation  tools  in  the  ATC  workplace  will  lead  to  an  increase 
in  ATCS  effort  while  adjusting  to  the  automation  and  attempting  to  incorporate  it  into  the  regular 
ATC  tasks.  During  the  introduction  of  automation  tools,  ATCSs  will  not  yet  possess  automated 
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behaviors  that  would  make  the  use  of  the  automation  fast  and  effortless.  When  using  the 
automation  during  that  learning  phase,  its  use  will  require  extra  effort,  resulting  in  an  increased 
workload.  We  remind  the  reader  that  we  required  that  ATCSs  did  not  have  previous  exposure  to 
the  DST.  Although  we  gave  them  training  and  hands-on  practice  for  several  days,  this  was  not 
enough  to  create  automated  behaviors.  The  increase  in  workload  with  full  automation  may  be  an 
indication  of  that.  In  practical  terms,  we  expected  one  of  two  things  to  happen.  Most  likely,  in 
high  traffic  task  load  situations,  ATCSs  would  revert  to  their  primary  goal  (safety)  and  leave  the 
automation  unused.  During  our  experiment,  we  did  observe  that.  However,  even  with  ATCSs 
using  the  tool  less  in  high  task  load  situations,  the  perceived  workload  increased.  The  new 
format  of  presentation  required  ATCSs  to  consciously  retrieve  the  information  from  the 
automation  tool  requiring  increased  effort.  The  alternative  scenario  was  that  ATCSs  would 
attempt  to  use  the  automation  tool  to  the  best  of  their  ability  even  in  high  task  load  situations. 
This  would  lead  to  larger  increases  in  workload.  Depending  on  the  workload  present,  because  of 
the  traffic  situation,  this  may  distract  (especially  the  D-side)  from  their  regular  duties.  Questions 
arise  as  to  how  long  this  increase  in  workload  lasts  for  novice  DST  users  and  whether  workload 
eventually  decreases  once  the  ATCSs  become  expert  DST  users.  The  data  from  the  current  study 
do  not  address  these  issues. 

Our  manipulation  of  task  load  clearly  resulted  in  an  increase  of  perceived  workload.  ATCSs, 
however,  rated  their  workload  to  be  moderate  under  high  task  load  conditions.  During  the 
experiment,  we  instructed  ATCSs  on  how  to  rate  the  instantaneous  subjective  assessment  of  their 
workload.  To  help  them  rate  their  workload  consistently,  we  provided  operational  anchors  with 
several  of  the  AT  WIT  values  (Appendix  F).  At  the  low  end  of  the  scale,  ATCSs  should  be  able 
to  accomplish  all  ATC  tasks  easily.  However,  at  the  high  end  of  the  scale,  ATCSs  would  have  to 
leave  some  of  the  ATC  tasks  unfinished.  During  the  high  task  load  scenarios,  we  observed 
ATCSs  controlling  traffic  in  a  manner  that  reflects  the  high  end  of  the  ATWIT  scale  (i.e.,  some 
of  the  tasks  remained  unfinished).  This  could  manifest  itself  in,  for  example,  the  late  initiation  or 
acceptance  of  assuming  a  strategy  that  reflects  controlling  for  safety  rather  than  elegance.  The 
fact  that  ATCSs  indicated  that  even  under  high  task  load  scenarios,  their  perceived  workload  was 
only  moderate  shows  an  underestimation  of  their  actual  workload.  The  reader  should  keep  in 
mind  that  our  analyses  focused  on  the  differences  between  task  load  levels  within  each  ATCS. 
The  graphical  representations  show  the  average  ratings  between  participants.  We  know  from  the 
raw  data  and  our  exit  debriefing  sessions  that  some  of  the  individual  ATCSs  rated  workload  to  be 
very  high.  The  task  load  levels  we  implemented  in  the  experimental  scenarios  were  higher  than 
what  ATCSs  currently  experience  in  positions  staffed  with  an  R-  and  D-side  team.  In  addition, 
the  six-participant  data  set  showed  a  stronger  effect  for  task  load  for  R-side  ATCSs.  The  level  of 
task  load  affected  R-side  ATCSs  more  directly  than  D-side  ATCSs.  The  R-side  ATCSs  were 
actively  controlling  and  communicating  with  traffic,  thus  increases  in  task  load  directly  had  an 
impact  on  the  number  of  actions  they  needed  to  perform,  which  was  then  reflected  in  higher 
workload  ratings.  Willems  and  Truitt  (1999)  found  lower  workload  ratings  under  monitoring 
conditions  than  active  control.  We  would  then  expect  that  for  the  D-side  ATCS,  only  a  slight 
difference  in  workload  ratings  should  be  seen  with  increasing  task  load. 
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Workload  ratings  gradually  decreased  for  the  low  task  load  scenarios;  however,  ratings  for  the 
high  task  load  scenarios  remained  relatively  constant  over  time.  As  the  scenario  progressed, 
ATCSs  had  an  opportunity  to  “catch  up”  after  the  initial  scenario  start  and,  because  task  load  was 
low,  ATCSs  had  a  chance  to  preplan  better  and  stay  ahead  of  the  traffic,  which  led  to  perceptions 
of  less  workload.  In  contrast,  the  high  task  load  scenarios  did  not  give  ATCSs  as  much  of  a 
chance  to  catch  up  and  preplan. 

When  sitting  on  the  R-side,  ATCSs  responded  faster  to  the  ATWIT  device  than  they  did  sitting 
on  the  D-side;  this  effect  was  stronger  in  the  low  task  load  conditions  than  in  the  high  task  load 
conditions.  Several  reasons  may  account  for  this  effect.  Although  the  ATWIT  device  was 
located  immediately  to  the  left  of  the  D-side  position,  when  controlling  traffic,  the  D-side  ATCSs 
were  physically  sitting  between  their  position  and  the  R-side  position.  When  ATWIT  went  off, 
the  D-side  ATCS  then  had  to  maneuver  closer  to  the  ATWIT  to  respond,  but  the  R-side  ATCS 
only  needed  to  quickly  reach  up  and  hit  it.  In  addition,  R-side  ATCSs  may  have  responded  faster 
to  the  ATWIT  because  they  wanted  to  focus  on  their  task  of  controlling  traffic.  By  answering 
ATWIT  quickly,  they  could  continue  with  their  task  and  not  need  to  devote  any  cognitive 
resources  to  remember  to  hit  it  later  (although,  they  were  instructed  to  immediately  respond  to 
ATWIT  when  it  went  off).  In  contrast,  D-side  ATCSs,  while  performing  their  duties,  may  have 
given  more  cognitive  resources  to  recall  that  ATWIT  went  off  and  finished  their  duties  before 
responding.  The  gradual  decrease  for  low  task  load  and  the  constant  level  of  workload  for  high 
task  load  clearly  reflects  this. 

3.2.2  NASA  Task  Load  Index 

The  NASA  TLX  consisted  of  six  questions  that  asked  for  ratings  of  mental,  physical,  and 
temporal  demands  as  well  as  performance,  effort,  and  frustration  levels  (Hart  &  Staveland, 

1988). 

3 .2.2. 1  Data  Analysis  and  Results 

A  2  x  2  x  3  (position  x  task  load  x  automation)  within-subjects  MANOVA  examined  the  effects 
of  the  experimental  variables  across  the  set  of  NASA  TLX  items.  We  provide  the  means,  SDs, 
MANOVA,  and  ANOVA  tables  in  Appendix  T.  We  found  a  significant  effect  for  task  load  [A  = 
.04,  F(6, 1 0)  =  42.14,p  <  .0001,  Appendix  T,  Table  T-2]  and  a  significant  position  x  task  load 
interaction  [A  =  .14,  F(6,10)  =  9.87,  p  <  .01,  Table  T-2],  Multivariate  analyses  of  the  simple 
effects  of  task  load  within  position  showed  a  stronger  effect  of  task  load  for  the  D-side  position 
[A  =  .01,  F(6,10)  =  99.50,  p  <  .0001,  A  =  .09,  F(6,10)  =  15.93 ,p  <  .001  for  the  R-side  and  D- 
side,  respectively,  Table  T-2],  Position  within  task  load  indicated  a  stronger  effect  for  position 
under  low  task  load  conditions  than  high  task  load  conditions  [A  =  .01,  F(6,10)  =  124.45,  p  < 
.0001  =  .12,  F(6,10)  =  12.77,  p  <  .001,  respectively,  Table  T-2],  Because  of  the  significant 

task  load  effect  and  position  x  task  load  interaction,  we  performed  subsequent  ANOVAs  for  each 
item  and  used  an  alpha  set  at  .0085  to  determine  significance. 

ATCSs  rated  the  mental,  physical,  and  temporal  demands  of  the  scenarios  and  their  levels  of 
effort  and  frustration  higher  when  controlling  traffic  from  the  high  task  load  scenarios  than  the 
low  task  load  scenarios  [F(l,15)  =  148.82,  30.20,  107.62,  33.89,  and  49.60,  respectively,  all  at 
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P  <  0001>  Figure  39,  Table  T-3,  Table  T-4,  Table  T-5,  Table  T-7,  and  Table  T-8,  respectively], 
ATCSs  rated  performance  lower  when  traffic  was  from  the  high  task  load  condition  than  the  low 
task  load  condition  [F(l,15)  =  21.51  ,p<  .001,  Table  T-6].  The  analyses  also  showed  several 
secondary  trends  in  the  data.  ATCSs  experienced  higher  levels  of  temporal  demand  under  full 
automation  (Figure  40).  ATCSs  rated  their  performance  better  when  on  the  R-side  than  the  D- 
side  (Figure  41).  As  the  level  of  automation  increased,  the  ratings  for  performance  decreased. 
Further,  the  effect  of  automation  was  stronger  under  high  task  load  conditions  (Figure  42). 


Mental  Physical  Temporal  Performance  Effort  Frustration 

Demand  Demand  Demand 

□  Low  Task  Load  ■  High  Task  Load 


Figure  39.  NASA  TLX  items  by  task  load. 
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Figure  40.  Temporal  demand  by  automation. 
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Figure  41 .  Performance  by  position. 
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Figure  42.  Performance  by  task  load  and  automation. 

3.2.2.2  Discussion 

Task  load  modified  the  effect  of  automation.  ATCSs  rated  their  performance  relatively  equal  in 
the  low  task  load  condition  regardless  of  the  degree  of  automation.  In  contrast,  when  working 
with  high  task  loads  of  traffic,  ATCSs  felt  their  performance  decreased  when  full  automation 
was  present.  The  ATCSs  may  have  felt  their  performance  decreased  when  full  automation  was 
present  because,  with  full  automation,  the  D-side  ATCSs’  responsibilities  differed  from  their 
normal  task  requirements,  and  this  may  have  altered  their  perception  of  their  performance 
quality.  In  addition,  the  use  of  full  automation  changed  the  ATCSs’  usual  tasks  and  replaced 
them  with  slightly  different  tasks  of  monitoring  the  automation  and  interpreting  the  information. 
The  ATCSs  did  not  have  a  frame  of  reference  for  completing  these  new  tasks  and,  when  task 
load  was  high,  believed  they  performed  worse.  During  the  debriefings,  ATCSs  mentioned  that 
the  use  of  the  DST  took  them  away  from  their  D-side  duties  and  out  of  the  picture.  It  may  also 
imply  that  ATCSs  have  a  tendency  to  underestimate  workload  levels. 

The  manipulation  of  task  load  had  a  strong  impact  on  the  NASA  TLX  items.  Results  indicated 
that  increases  in  task  load  led  to  higher  ratings  for  mental,  physical,  and  temporal  demand,  effort, 
and  frustration.  ATCSs  stated  their  performance  was  worse  when  controlling  traffic  from  the 
high  task  load  scenarios.  We  expected  higher  ratings  for  workload  measures  with  higher  levels 
of  traffic.  At  the  volumes  of  traffic  presented  in  the  high  task  load  scenarios,  the  ATCSs  had  a 
constant  push  of  traffic  that  did  not  allow  for  them  to  “catch  up”  and  preplan  actions.  This 
would  affect  perceived  workload  along  the  TLX  items. 

Besides  task  load  and  automation,  position  affected  self-reported  performance  ratings.  ATCSs 
felt  they  controlled  traffic  better  when  on  the  R-side  in  comparison  to  the  D-side. 
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3.2.3  Post-Scenario  Questionnaire 


3.2.3 . 1  Data  Analysis  and  Results 

We  conducted  a  2  x  2  x  3  (position  x  task  load  x  automation)  within-subjects  ANOVA  on  the 
item  asking  ATCSs  how  hard  they  were  working  during  the  scenario  (Appendix  U).  Automation 
changed  ATCS  perception  of  how  hard  they  worked  [F(2,30)  =  3.51  ,p<  .05,  Figure  43, 
Appendix  U,  Table  U-l],  Tukey  HSD  post  hoc  tests  showed  that  ATCSs  stated  they  worked 
harder  when  full  automation  was  present  than  when  they  had  no  automation.  ATCSs’ 
perceptions  of  working  hard  were  not  statistically  different  between  the  full  and  limited 
automation  levels,  or  between  the  limited  and  no  automation  conditions.  Task  load  or  position 
did  not  influence  their  perceptions  of  working  hard  in  the  full  data  set.  However,  in  the  data  set 
composed  of  the  last  6  participants,  ATCSs  described  working  harder  under  high  task  load 
conditions  than  under  low  task  load  conditions  [F(l,5)  =  29.91  ,p<  .01,  Figure  44,  Table  U-2]. 


— ♦ — Low  Task  Load  . a  High  Task  Load 

Figure  43.  Working  hard  by  automation. 
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Low  Task  Load  High  Task  Load 


Figure  44.  Working  hard  by  task  load  (N=6). 
3.2.3.2  Discussion 


ATCSs  described  working  harder  when  they  used  full  automation  relative  to  no  automation. 

This  coincides  with  other  researchers  findings  (e.g.,  Wiener,  1985)  that  the  use  of  automation 
often  leads  to  perceptions  of  increased  workload.  The  automation  requires  the  user  to  monitor  it 
and  still  perform  their  other  tasks,  in  this  case  assisting  the  R-side  ATCS.  Researchers  have 
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documented  that  user  monitoring  of  the  system  influences  perceptions  of  higher  workload  (e.g., 
Becker  et  al.,  1991).  Of  interest  though,  is  that  the  self-reported  workload  ratings  under  full 
automation  were  only  moderate.  This  would  imply  that  ATCSs  still  have  resources  to  devote  to 
other  tasks  and  that  the  introduction  of  automation  did  not  overburden  them. 

Results  from  the  six-participant  data  set  showed  higher  ratings  for  working  hard  when  ATCSs 
controlled  traffic  from  the  high  task  load  scenarios.  This  would  be  expected.  Higher  volumes  of 
traffic  require  more  control  actions,  increasing  workload.  Position  did  not  affect  the  perception 
of  working  hard. 

3.3  Eye  Movements 

Air  traffic  control  is  a  visually  demanding  task.  ATCSs  continuously  monitor  the  radar  display, 
flight  progress  strips,  and  other  information  displays  for  information  that  may  require  them  to 
take  action.  The  data  gained  from  the  visual  scanning  task  (i.e.,  the  monitoring  and  processing  of 
visual  information)  forms  the  ATCS's  main  source  of  information.  The  ATCS  integrates 
information  obtained  through  radio  communications,  phone  lines,  his  or  her  knowledge  of  the 
airspace,  procedures,  and  other  sources  with  that  obtained  from  the  radar  display  and  other  visual 
displays.  To  obtain  data  from  visual  displays,  the  ATCS  scans  them  for  new  data  or  for 
situations  that  require  examination  that  is  more  careful.  The  ATCS  eye  movements  during  the 
visual  scanning  task  consist  of  stationary  periods  or  fixations,  jumps  between  fixations  or 
saccades,  and  eye  blinks  (Willems  et  al.,  1999). 

Rotting  (1999)  reported  fixation  durations  for  reading  tasks  that  averaged  225  ms.  Fixation 
durations  for  image  observations  and  search  tasks  averaged  330  ms  and  250  ms,  respectively. 
Although  extensive  research  is  available  on  eye  movement  characteristics  in  reading,  the  ATC 
task  is  more  than  reading  alone.  The  long  fixation  durations  of  500-600  ms  in  ATC  (Willems  et 
al.,  1 999)  clearly  reflects  this. 

In  previous  studies  at  the  RDHFL,  researchers  used  eye  movement  characteristics  to  measure 
changes  in  visual  scanning  behavior  as  a  function  of  experimental  conditions  (Stein,  1992; 
Willems  et  al.,  1999;  Willems  &  Truitt,  1999).  Willems  et  al.  were  the  first  to  use  head-mounted 
oculometry  synchronized  with  dynamic  ATC  events.  They  were  able  to  calculate  the  eye 
movement  characteristics  of  participants  looking  at  the  radar  display  versus  other  data  displays 
and  aircraft  versus  other  objects.  Fixations  on  aircraft  representations  were  substantially  longer 
than  on  any  other  object.  These  researchers  concluded  that  it  is  likely  that  more  information 
processing  takes  place  during  the  longer  eye  movement  fixations  on  aircraft  representations. 

Willems  and  Truitt  (1999)  implemented  several  measures  derived  from  the  conditional 
information  index,  first  introduced  by  Ellis  (1986).  Willems  and  Truitt  adapted  the  conditional 
information  index  to  investigate  the  randomness  of  the  visual  scanning  distribution.  Eye 
movement  characteristics  did  not  change  as  a  function  of  involvement  or  task  load.  However, 
when  the  ATCS  actively  controlled  traffic  under  high  task  load,  the  visual  scan  was  less 
structured  than  when  the  ATCS  passively  monitored  traffic.  The  results  indicate  that  ATCSs 
maintain  a  better  scan  of  traffic  when  they  actively  control  it.  At  the  outset,  ATCSs  rely  more  on 
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relief  briefing  information,  and,  therefore,  they  can  manage  with  shorter  fixations.  Once 
controlling  traffic,  the  ATCS  may  need  to  fixate  longer  in  order  to  evaluate  the  relevance  of  the 
displayed  information. 

The  visual  scanning  measures  in  the  present  study  focused  on  the  D-side  ATCS.  In  this 
experiment,  we  implemented  the  R-side  and  D-side  ATCS  positions.  Our  previous  experiments 
have  limited  themselves  to  R-side  ATCSs.  The  roles  and  responsibilities  of  the  D-side  ATCS  are 
quite  different  from  those  of  the  R-side  ATCS  and  will  vary  from  ARTCC  to  ARTCC.  The  D- 
side  ATCS  functions  as  a  second  pair  of  eyes  that  assist  the  R-side  ATCS  in  looking  for 
information  on  the  radar  display.  This  would  suggest  that  under  normal  circumstances,  the  D- 
side  ATCS  should  display  similar  visual  scanning  behavior  when  looking  at  the  radar  display  as 
we  have  previously  found  for  R-side  ATCSs  (e.g.,  Willems  &  Truitt,  1999).  The  D-side  ATCS, 
on  the  other  hand,  also  has  responsibilities  that  will  move  his/her  attention  away  from  the  radar 
display.  With  the  use  of  automation  tools  such  as  the  one  used  in  this  experiment,  one  would 
expect  the  D-side  ATCS  to  spend  more  time  away  from  the  radar  display,  attending  to  and 
resolving  more  strategic  problems  using  the  DST. 

3.3.1  Data  Analysis  and  Results 

In  this  experiment,  we  used  operational  hardware  and  software  to  replicate  the  ATCS  operational 
environment.  Therefore,  we  developed  software  to  record  visual  scanning  related  data.  We 
collected  two  redundant  data  sources.  The  first  data  set  consisted  of  the  SAR  tapes  collected 
from  the  Host  Computer  System.  The  second  data  set  consisted  of  Extensible  Markup  Language 
(XML)  recorded  by  our  middleware  (Appendix  E).  These  XML-files  contained  several 
messages  normally  available  on  the  DSR  token  ring  that  we  needed  to  determine  the  location  and 
size  of  objects  displayed  on  the  radar  display.  Our  programmers  designed  a  conversion  utility  in 
JAVA  that  transformed  the  XML  data  into  binary  files  that  conformed  to  the  format  of  our  eye 
movement  integration  software  (ITAP  developed  by  programmers  at  the  RDHFL).  Our 
programmers  also  developed  software  that  recorded  which  windows  the  DST  displayed  on  the 
D-side  display.  The  software  collected  the  name,  the  location,  and  the  size  of  each  window  and 
time  stamped  the  data.  We  used  the  in-house  developed  ITAP  software  to  integrate  simulation 
and  eye  movement  data.  The  ITAP  program  uses  the  converted  XML  files,  the  data  recorded 
from  X-event  information  from  the  DST  display,  and  the  eye  movement  data.  ITAP  determines 
with  which  scene  plane  the  line  of  sight  intersects  and  whether  or  not  a  sample  belongs  to  a 
fixation,  a  saccade,  or  a  blink  (Figure  45). 

The  oculometer  data  formed  the  basis  for  the  calculated  variables  related  to  visual  scanning. 
Visual  scanning  targets  included  radar  display,  keyboard  area,  ATWIT  device,  flight  strip  bays, 
overhead  sector  maps,  communication  panels,  Computer  Readout  Device  (CRD),  and  the  DST 
windows.  Appendix  V  contains  descriptions  that  are  more  detailed  and  information  about  the 
computation  of  the  visual  scanning  DVs. 
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Figure  45.  Eye  movement  data  reduction  and  analysis  process. 

We  reduced  the  raw  visual  scanning  data.  We  expressed  it  as  general,  scene,  object,  or  structure 
based  eye  movement  characteristics  and  conducted  appropriate  analyses  (Table  7).  General  eye- 
movement  characteristics  included  fixations,  saccades,  and  blinks.  Fixation  characteristics 
included  time  of  onset,  duration,  the  plane  being  looked  at,  the  area  covered  by  small  eye 
movements  within  the  fixations,  and  the  coordinates  relative  to  the  plane.  Saccade 
characteristics  included  information  on  the  number  and  magnitude  of  the  saccade  and  the 
average  velocity  during  the  saccade.  Blink  characteristics  included  number,  mean  duration,  and 
mean  distance.  We  integrated  the  eye  movement  data  with  the  DST  information  for  the  scene 
based  eye  movement  data.  This  included  fixations  and  saccades  for  each  scene  plane  within  the 
experiment,  as  mentioned  previously.  Object  based  eye  movement  data  included  aircraft 
position  symbol  and  Full  Data  Blocks  (FDBs)  from  the  radar  display.  Finally,  structure  of  eye 
movement  characteristics  included  conditional  information,  i.e.,  the  predictability  of  the  target  or 
location  of  a  fixation  when  we  have  information  about  the  previous  fixation.  Object-based 
conditional  information  uses  the  probability  that  a  fixation  on  object  A  (e.g.,  USA123)  is 
followed  by  a  fixation  on  object  B  (e.g.,  TWA46)  and  weighs  that  probability  with  the 
probability  that  a  fixation  fell  on  object  A.  Range-based  conditional  information  extends  the 
objects  based  principle.  It  divides  the  fixations  into  bins  based  on  the  distance  from  the  next 
fixation  and  uses  the  probability  that  after  a  fixation  landing  in  bin  A,  the  next  fixation  falls  in 
bin  B.  For  box-based  conditional  information,  we  divide  the  radar  screen  into  a  grid  of  10  x  10 
and  calculate  probabilities  for  each  cell  of  that  grid.  Finally,  the  ring-based  conditional 
information  is  more  applicable  to  terminal  environments  because  it  requires  us  to  divide  the  radar 
screen  into  concentric  rings  around  the  center  of  the  radar  display.  Appendix  W  contains  the 
MANOVA,  ANOVA,  and  means  and  SDs  tables. 
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Table  7.  Visual  Scanning  Analyses 


Category 

Visual  Scanning  Characteristics 

Type  of  Analysis 

New  IVs  and  Levels 

Scene  based 

Radar  Display  and  DST  Display 

Fixations  (number,  percent,  mean  duration) 
Saccades  (number,  percent,  mean  distance) 

2x2x3  (task  load  x 
display  x  automation) 
MANOVAs  and 
ANOVAs 

Display:  Radar 
display  vs.  DST 
display 

Object  based 

Aircraft  Position  Symbol  and  FDB 
(number,  duration) 

2  x  2  x3  (task  load  x 
information  x 
automation) 
MANOVAs  and 
ANOVAs 

Information:  Aircraft 
Position  Symbol  vs. 
FDB 

General 

Fixations  (number,  mean  duration,  mean  area, 
visual  efficiency) 

Saccades  (number,  mean  duration,  mean 
distance,  eye  motion  workload) 

Blinks  (number,  mean  duration,  mean 
distance) 

2x3  (task  load  x 
automation) 
MANOVAs  and 
ANOVAs 

Structure 

Object-based  conditional  information  index 
Range-based  conditional  information  index 
Box  based  (screen  is  divided  into  10  xlO 
grid)  conditional  information  index 

Ring  based  conditional  information  index 

2x3  (task  load  x 
automation) 
MANOVAs  and 
ANOVAs 

| 

Descriptive 

Visual  scanning  target  (D-side 
communications  panel,  R-side 
communications  panel,  D-side  map,  R-side 
map,  CRD,  ATWIT,  flight  strip  bay  1,  flight 
strip  bay  2) 

DST  windows  (clock,  aircraft  list,  graphic 
plans  display,  response  display,  trial  plans, 
plans  display,  other  displays) 

Means  and  SDs 

We  screened  the  visual  scanning  data  based  on  fixation  characteristics  to  remove  any  outlier  data 
points.  To  do  so,  we  used  a  multivariate  approach  that  tested  the  Mahalanobis  distance  statistic. 
The  Mahalanobis  distance  is  the  distance  of  a  point  from  the  centroid  of  a  multidimensional 
space  based  on  the  IVs  (StatSoft,  2000).  We  tested  a  given  data  point  value  against  a  critical 
value  and  plotted  the  values  to  determine  whether  any  of  the  observations  were  outliers.  In  all, 
we  had  three  observations  that  were  outliers  and  removed  them  from  further  analysis. 

3.3.1 .1  Scene-Based  Eye  Movement  Characteristics 

The  eye  movement  data  integration  software  calculated  fixations  and  saccades  on  each  of  the  10 
scene  planes  examined  in  this  study.  Due  to  the  manipulation  of  automation,  we  were  most 
interested  in  the  D-side  ATCS’s  visual  scanning  of  the  DST  display  and  the  radar  display.  We 
created  an  additional  IV  (display  type)  and  conducted  a  2  x  2  x  3  (task  load  x  display  x 
automation)  within-subjects  MANOVA.  For  the  other  scene  planes  and  DST  windows,  we  only 
conducted  descriptive  statistics  (Appendix  W). 
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The  MANOVA  for  the  number,  percent,  and  mean  duration  of  fixations  showed  significant 
effects  for  display  type,  automation  and  the  display  type  x  automation  interaction  [A  =  .39, 
F(3,13)  =  6.12, p  <  .01,  A  =  .14,  F(6,10)  =  10.28,/?  <  .001  and  A  =  .07;  F(6, 10)  =  20.81  ,p  < 
.0001,  Appendix  W,  Table  W-38,  respectively].  We  used  an  adjusted  alpha  of  .017  for 
subsequent  follow-up  ANOVAs. 

The  ANOVA  for  the  number  of  fixations  showed  significant  effects  for  display  type  and 
automation  and  the  display  type  x  automation  interaction  [F(l,15)  =  5.87,/?  <  .05;  F(2,30)  = 
24.54,/?  <  .0001;  F( 2,30)  =  50.07,/?  <  .0001,  Table  W-39,  respectively].  Because  there  was  a 
significant  interaction  between  display  type  and  automation,  we  explain  the  main  effects  of  these 
variables  within  the  interaction.  The  simple  effect  of  display  type  was  significant  for  the  no  and 
full  automation  conditions  [F(l,15)  =  74.18,/?  <  .0001  and  F(l,15)  =  19.14,  p  <  .01,  Table  W-43, 
respectively]  but  not  for  the  limited  automation  condition.  Under  no  automation,  ATCSs  fixated 
more  on  the  radar  display,  whereas,  under  full  automation,  they  fixated  more  on  the  DST  display 
(Figure  46).  Please  note  that  for  the  purposes  of  the  analyses,  we  did  not  remove  any  fixations  or 
saccades  on  the  SAVANT  queries,  which  occurred  on  the  same  display  as  the  DST.  In  other 
words,  there  were  fixations  and  saccades  on  the  DST  even  when  the  DST  was  not  in  use;  these 
fixations  and  saccades  are  reflected  by  the  none  zero  values  in  the  no  automation  conditions. 
These  fixations  on  SAVANT  are  consistent  across  all  conditions  and  represent  a  systematic  error 
or  bias  that  is  not  dependent  upon  a  particular  condition. 


Figure  46.  Number  of  scene  plane  fixations  by  display  and  automation. 

For  the  percent  of  total  number  of  fixations,  we  found  a  significant  effect  for  automation  and  the 
display  type  x  automation  interaction  [F(2,30)  =  29.09,/?  <  .0001  and  F( 2,30)  =  53.23,/?  <  .0001, 
Table  W-40],  The  simple  effect  of  display  type  was  significant  under  the  no  and  full  automation 
conditions  [F(l,15)  =  70.34,/?  <  .0001  and  F(l,15)  =  143.95,/?  <  .001,  Table  W-40, 
respectively],  but  not  for  the  limited  automation  condition.  As  with  the  number  of  fixations, 
ATCSs  spent  a  higher  percentage  of  fixations  on  the  radar  display  when  no  automation  was 
present,  but  they  spent  a  higher  percentage  of  fixations  on  the  DST  display  when  automation  was 
present  (Figure  47).  The  percentage  of  fixations  on  the  displays  did  not  differ  under  limited 
automation  conditions.  We  ask  the  reader  to  note  that  these  percentages  do  not  add  up  to  100 
because  we  include  only  those  fixations  on  the  radar  or  DST  display,  and  excluding  fixations 
elsewhere  (e.g.,  sector  maps,  CRD).  If  we  included  all  fixations  on  all  scene  plane  surfaces,  we 
would  get  a  percentage  that  adds  to  100. 
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Radar  Display  —a—  DST  Display 


Figure  47.  Percent  of  fixations  by  display  and  automation. 

The  effect  of  automation  and  the  display  type  x  automation  interaction  was  significant  for  the 
mean  duration  of  fixations  [F(2, 30)  =  11.34,  p  <  .0001  and  F(2, 30)  =  29.12,p  <  .0001,  Table 
W-41 ,  respectively].  The  type  of  display  influences  the  main  effect  of  automation.  The  two-way 
interaction  between  display  type  and  automation  showed  that  the  simple  effect  of  display  type 
was  significant  for  the  no  and  full  automation  conditions  [F(l,15)  =  27.02, p  <  .0001,  F{  1,1 5)  = 
4-54 ,p<  .05,  respectively,  Table  W-41]  but  not  for  the  limited  automation  condition.  The  mean 
duration  of  fixations  was  longer  on  the  radar  display  than  the  DST  display  when  no  automation 
was  present  (Figure  48).  In  contrast,  under  full  automation,  the  mean  duration  of  fixations  was 
longer  on  the  DST  display  (Figure  48). 


— Radar  Display  — B — DST  Display 

Figure  48.  Mean  duration  of  fixations  by  display  and  automation. 

The  MANOVA  examining  the  number,  percent,  and  distance  of  saccades  showed  significant 
effects  for  task  load,  display  type,  and  automation,  and  the  interactions  between  task  load  x 
automation,  display  type  x  automation  were  significant  [A  =  .26,  F(3,13)  =  12.44,  p  <  .001,  A  = 
.14;  F(3,13)  =  25.87 ,/?  <  .0001;  ^  =  .13,  F(6,10)  =  1 1.31  ,p<  .001;  A  =  .29,  F(6,10)  -  4.12, 

P  <  -05,  A  =  .07,  and  F(6,10)  =  20.66,  p  <  .0001,  Table  W-42,  respectively].  We  used  an 
adjusted  alpha  of  .017  for  the  follow-up  univariate  analyses. 

The  univariate  results  for  the  number  of  saccades  showed  a  significant  effect  for  automation  and 
a  significant  interaction  between  display  type  and  automation  [F(2,30)  =  21.57,/?  <  .001  and 
F(2,30)  =  49.24,/?  <  .0001,  Table  W-43,  respectively].  The  display  type  influenced  the  effect  of 
automation  on  number  of  saccades.  The  simple  effect  of  display  type  within  both  the  no  and  full 
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automation  conditions  was  significant  [F(l,15)  -  61.72, p  <  .0001  and  F(l,15)  =  35.0903, 
p  <  .0001,  Table  W-43,  respectively].  The  number  of  saccades  on  the  radar  display  was  higher 
under  no  automation  conditions,  while  the  number  of  saccades  on  the  DST  display  was  higher 
under  full  automation  (Figure  49). 
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Figure  49.  Number  of  saccades  by  display  and  automation. 

The  ANOVA  for  the  percent  of  saccades  spent  on  the  two  displays  showed  a  significant  effect 
for  automation  and  the  display  type  x  automation  interaction  [F(2,30)  =  30.65 ,p  <  .0001  and 
F(2,30)  =  57.9 6,p  <  .0001,  Table  W-44,  respectively].  The  type  of  display  qualified  the  effect 
of  automation.  The  simple  effect  of  display  type  was  significant  for  both  the  no  and  full 
automation  conditions  [F(l,15)  =  60.84, p  <  .0001,  and  F(l,15)  =  39.68 ,p  <  .0001,  Table  W-44, 
respectively]  but  not  for  the  limited  automation  condition.  The  percent  of  saccades  on  the  radar 
display  was  higher  than  that  on  the  DST  display  under  no  automation;  in  contrast,  the  percent  of 
saccades  on  the  DST  display  was  higher  under  full  automation  conditions  (similar  trend  to  Figure 
49). 

For  saccade  distance,  the  ANOVA  showed  significant  effects  for  task  load,  display  type,  task 
load  x  automation  [F(l,15)  =  23.64,/?  <  .001;  F(l,15)  =  79.06,/?  <  .0001;  ,F(2,30)  =  6.37, 
p  <  .01,  Table  W-45,  respectively]  and  secondary  trends  for  automation,  display  type  x 
automation,  and  task  load  x  display  type  x  automation.  We  explain  all  effects  within  the  context 
of  the  three-way  interaction.  The  simple  effects  of  display  type,  automation  and  the  display  type 
x  automation  interaction  were  significant  under  low  task  load  [F(l  ,1 5)  =  36.34,  p  <  .01 ;  F( 2,30) 

=  6.89,/?  <  .01;  F(2,30)  =  6.98,/?  <  .01,  Table  W-45,  respectively].  Under  low  task  load 
conditions,  mean  saccade  distance  was  higher  for  the  DST  display  and  no  automation  compared 
to  limited  automation.  Mean  saccade  distance  was  longest  under  the  no-automation  condition 
and  the  DST  display  (Figure  50).  Only  the  effect  of  display  type  was  significant  under  high  task 
load  conditions  [F(l,15)  =  77.27,/?  <  .0001,  Table  W-45].  When  task  load  was  high,  saccade 
distance  was  longest  on  the  DST  display  (Figure  51). 
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Figure  50.  Mean  saccade  distance  under  low  task  load  by  display  and  automation. 
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Figure  51.  Mean  saccade  distance  under  high  task  load  by  display  and  automation. 

For  the  analyses,  we  used  DST  display  as  one  level  of  the  display  information  variable.  These 
analyses  included  any  visual  scanning  hits  on  the  DST  display.  However,  the  DST  presented 
information  to  the  D-side  ATCS  in  several  windows  for  which  we  have  fixation  characteristics. 
These  windows  include  the  aircraft  list,  graphic  plans  display,  trial  plans  display,  plans  display, 
clock,  and  so  on.  Based  on  the  level  of  automation,  we  instructed  ATCSs  to  access  certain 
windows  only.  In  the  no-automation  condition,  we  did  not  display  the  DST  windows.  For 
limited  automation,  ATCSs  could  view  the  aircraft  list,  whereas,  in  full  automation,  all  DST 
windows  were  available  for  viewing.  Please  note  that  the  visual  scanning  values  for  the  limited 
automation  condition  are  somewhat  inflated  for  three  reasons.  First,  ATCSs  occasionally  and 
accidentally  opened  additional  windows  such  as  the  graphic  plans  display  during  the  limited 
automation  conditions.  We  could  not  modify  the  DST  to  inhibit  ATCSs  from  opening  windows 
other  than  the  aircraft  list  in  the  limited  automation  condition.  Instead,  the  ATC  SME  reminded 
ATCSs  to  immediately  close  windows  that  they  accidentally  opened.  In  addition,  the  grand 
mean  substitution  we  used  for  replacement  of  missing  data  inflates  the  values.  The  SAVANT 
responses  are  also  included  in  these  values,  as  they  are  for  the  full  automation  condition.  We  set 
all  values  for  the  no  automation  condition  to  zero  because  the  DST  was  not  running  during  these 
scenarios.  Appendix  W  contains  the  means  and  SDs  for  these  windows.  Here,  we  only  highlight 
those  descriptive  statistics  that  are  of  interest.  As  shown  in  Figure  52,  ATCSs  fixated  on  the 
aircraft  list  more  than  on  the  graphic  plans  display  under  full  automation  and  did  not  take 
advantage  of  the  trial  plans  capability. 
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Figure  52.  Means  and  SDs  for  fixations  on  DST  windows. 

3 .3 . 1 .2  Object-Based  Eye  Movement  Characteristics 

The  object  based  eye  movement  characteristics  included  the  aircraft  position  symbol  and  the 
FDB  as  an  extra  IV  -  information.  We  conducted  a  2  x  2  x  3  (task  load  x  information  x 
automation)  within-subjects  MANOVA  on  the  number  and  mean  duration  of  fixations.  The 
MANOVA  showed  significant  effects  for  information,  automation,  the  task  load  x  information 
interaction,  and  the  information  x  automation  interaction  [A  =  .02,  F(2,14)  =  285.46,  p  <  .0001; 

A  =  .22,  F( 4,12)  =  10.86, p  <  .001;  A  =  .21,  F(2,14)  =  3.75, p  <  .05,  F(4,12)  =  11.12 ,p  <  .001, 
Table  W-47,  respectively].  We  used  an  adjusted  alpha  of  .025  for  the  follow-up  univariate 
analyses. 

The  ANOVA  for  the  number  of  object  based  fixations  showed  significant  effects  for  information, 
automation,  and  the  information  x  automation  interaction  [F(l,15)  =  209.58 ,p  <0001;  F( 2,30)  = 
24.965, p  <  .0001;  F(2,30)  =  15.57,  p  <  .0001,  Table  W-48,  respectively].  The  two-way 
interaction  between  display  and  automation  qualified  both  of  the  main  effects,  and  we  focused  on 
this.  The  simple  effect  of  information  within  the  no-,  limited-,  and  full-automation  conditions 
was  significant  [A(l,15)  =  209.22,;?  <  .0001;  F(l,15)  =  78.82,;?  <  .0001;  F(l,15)  =  153.66, p  < 
.0001,  respectively  Table  W-48]  but  not  for  the  full  automation  condition.  Although  more 
fixations  occurred  on  the  FDB,  automation  attenuated  this  effect.  As  the  degree  of  automation 
increased,  the  number  of  fixations  on  FDB  decreased  (Figure  53). 
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Figure  53.  Number  of  object  based  fixations  by  information  and  automation. 
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The  ANOVA  for  the  duration  of  object  based  fixations  showed  significant  effects  for 
information  and  task  load  x  information  [F(l,15)  =  20.85, p  <  .0001;  F(l,15)  =  5.64,/?  <  .05, 
respectively  Table  W-49].  For  the  two-way  interaction,  the  simple  effect  of  information  was 
significant  for  low  and  high  task  loads  [F(l,15)  =  203.56,/?  <  .0001  and  F(l,15)  =  123.50,/?  < 
.0001,  respectively  Table  W-49].  The  fixation  duration  was  longer  for  FDBs;  however,  low  task 
load  attenuated  this  (Figure  54). 
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Figure  54.  Fixation  duration  by  task  load  and  information. 

3.3.1 .3  General  Eye  Movement  Characteristics 

We  conducted  separate  analyses  for  each  of  the  general  eye  movement  characteristics  -  fixations, 
saccades,  and  blinks.  These  MANOVAs,  structured  as  2  x  3  (task  load  x  automation)  repeated 
measures  designs,  addressed  the  differences  across  scenarios.  The  MANOVA  for  fixations 
included  the  number,  mean  and  SDs  for  duration,  area,  and  visual  efficiency.  We  did  not  find 
significant  differences  at  the  multivariate  level;  however,  at  the  univariate  level,  several  trends 
existed.  A  trend  for  automation  on  the  mean  duration  of  fixations  showed  that  as  automation 
increased,  the  mean  duration  of  the  fixations  increased  (Figure  55,  Table  W-20).  As  automation 
increased,  the  SD  of  the  duration  of  the  fixations  increased  (Figure  56,  Table  W-21).  Secondary 
trends  in  the  data  showed  effects  for  task  load  and  automation  for  visual  efficiency.  As  task  load 
increased,  visual  efficiency  decreased  (Figure  57,  Table  W-24).  As  automation  increased,  visual 
efficiency  increased  (Figure  57,  Table  W-24). 
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Figure  55.  Mean  duration  of  fixations  by  automation. 
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Figure  56.  SD  of  fixation  durations  by  automation. 
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Figure  57.  Visual  efficiency  by  task  load  and  automation. 

The  MANOVA  for  saccades  examined  the  effects  of  task  load  and  automation  on  the  number, 
mean,  and  SD  of  durations,  mean  and  SD  of  distance  moved,  and  eye  motion  workload.  The 
effects  of  task  load  and  automation  did  not  reach  statistical  significance  at  the  multivariate  level, 
however,  univariate  results  showed  several  trends  in  the  data.  As  automation  increased,  the 
mean  distance  traveled  in  the  saccade  decreased  (Figure  58,  Table  W-27).  A  secondary  trend 
showed  that  as  task  load  increased,  the  number  of  saccades  increased  (Figure  59,  Table  W-26). 
As  automation  increased,  the  eye  motion  workload  indicated  that  the  eye  moved  less  per  second 
(Figure  60,  Table  W-38). 
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Figure  58.  Mean  distance  traveled  in  saccade  by  automation. 
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Figure  59.  Number  of  saccades  by  task  load. 
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Figure  60.  Eye  motion  workload  by  automation. 

The  MANOVA  for  blinks  examined  the  effects  of  the  experimental  variables  on  the  number  of 
blinks,  blink  duration,  and  distance.  No  effects  reached  statistical  significance  at  the  multivariate 
level  nor  were  there  any  trends  at  the  univariate  level  of  analysis. 


The  2x3  (task  load  x  automation)  within-subjects  MANOVA  examining  structure  of  eye 
movement  characteristics  showed  significant  effects  for  task  load  [A  =  .16,  F(4,12)  =  26.52,  p< 
.001,  Table  W-50].  We  used  an  adjusted  alpha  of  .013  for  subsequent  univariate  analyses. 


The  ANOVA  for  the  object  conditional  index  showed  a  significant  effect  for  task  load  [F(l,15)  = 
66.71,/?  <  .001,  Table  W-51],  The  structure  in  the  visual  scan  decreased  under  high  task  load 
conditions  (Figure  61,  Table  W-51).  The  ANOVA  for  box  conditional  information  showed  a 
significant  main  effect  for  task  load  [F(l,15)  =  5.34,/?  <  .05,  Table  W-53]  and  a  trend  for 
automation.  As  automation  and  task  load  increased,  the  structure  of  the  visual  scan  decreased 
(Figure  62). 
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Figure  61.  Object  conditional  index  by  task  load. 


Figure  62.  Box  conditional  information  by  task  load  and  automation. 

3.3.2  Discussion 

The  general  eye-movement  characteristics  indicate  that  fixations  were  longer  with  increasing 
levels  of  automation.  Based  on  the  results  from  previous  studies,  we  had  expected  that  the 
characteristics  of  fixations  on  information  on  the  radar  display  would  not  change  because  the 
stimulus  did  not  change,  and,  therefore,  the  mechanism  to  pick  up  information  from  the  stimulus 
would  not  change  either.  The  information  on  the  DST  was  either  quite  similar  to  that  on  the 
radar  display  (i.e.,  an  aircraft  representation  in  a  graphical  format)  or  similar  to  information  on  a 
flight  progress  strip  in  electronic  format.  If  anything,  we  expected  a  decrease  in  fixation 
duration. 

In  previous  studies,  we  tracked  eye  movements  of  an  R-side  ATCS.  We  had  made  the  null 
hypothesis  that  the  eye  movement  characteristics  of  an  ATCS  would  not  change  between  the  R- 
side  and  the  D-side  ATCS  position.  We  find,  however,  that  the  distribution  of  the  number  of 
fixations  between  the  position  symbols  and  the  data  blocks  of  aircraft  is  very  different  for  D-side 
ATCSs  compared  to  R-side  ATCSs.  In  our  experience  with  R-side  ATCSs,  they  fixated  on  data 
blocks  and  aircraft  positions  an  equal  number  of  times.  This  may  have  occurred  because  we 
used  a  spotlight  model  that  creates  a  2-in.  radius  around  the  center  of  a  fixation.  We  felt  that  the 
ATCS  then  absorbed  information  about  the  aircraft  as  a  whole  and  not  by  a  particular  piece  of 
information  (i.e.,  FDB  or  position  symbol).  This,  in  turn,  would  result  in  an  equal  number  of 
fixations  on  the  data  blocks  and  the  position  of  the  aircraft  closest  to  the  center  of  the  fixation.  In 
the  current  experiment,  we  find  a  very  different  distribution  of  fixations.  The  D-side  ATCS 
consistently  focused  more  on  the  FDBs.  Overall,  there  were  more  fixations  on  the  FDB  and 
aircraft  position  symbols  under  conditions  without  automation  because  the  D-side  ATCS  is  more 
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of  an  assistant  for  the  R-side  ATCS.  With  an  increase  in  automation,  the  number  of  fixations  on 
the  radar  display  decreases  (i.e.,  the  R-side  ATCS  will  have  less  assistance  of  the  D-side  ATCS 
in  scanning  the  radar  display  for  information). 

The  difference  in  fixation  distribution  for  the  D-side  ATCS  compared  to  the  findings  from  our 
previous  studies  may  also  indicate  a  different  strategy  used  by  the  D-side  ATCS.  The  D-side 
ATCS  focuses  less  on  aircraft  position  symbols  than  the  R-side  ATCS  in  previous  experiments. 
Cushing  (1989),  for  example,  suggests  that  ATCSs  have  a  process  that  checks  distances  between 
aircraft.  That  suggestion,  however,  can  still  result  in  a  pattern  in  which  an  ATCS  would  either 
fixate  on  two  aircraft  positions  sequentially,  fixate  on  a  location  close  to  both  aircraft  and  process 
the  position  information,  or  separate  the  fixations  on  the  positions  by  several  other  fixations  but 
process  the  position  information  after  having  retrieved  all  information  needed  to  check  the 
distance.  The  R-side  ATCS  has  a  tactical  function  with  separation  as  the  primary  goal.  Fixating 
on  aircraft  positions  allows  the  R-side  ATCS  to  determine  if  the  required  separation  standards 
are  met.  In  contrast,  the  D-side  ATCS  looks  less  at  aircraft  positions  but  looks  for  information  in 
the  data  blocks.  Although  processing  information  may  occur  in  areas  away  from  the  fovea  (i.e., 
further  away  from  the  center  of  the  fixation),  evidence  exists  that  the  visual  system  prefers 
processing  information  that  is  present  in  the  foveal  area.  The  shift  in  the  scanning  strategy 
between  the  R-side  ATCS  and  the  D-side  ATCS  seems  to  show  that  the  D-side  ATCS  is  less 
tactical  and  focuses  on  the  data  block  information  instead. 

One  of  the  concerns  about  the  data  analyses  and  results  of  our  visual  scanning  work  is  that  eye 
movements  are  the  outcome  of  possibly  several  underlying  cognitive  processes.  Viviani  (1990) 
suggests  that  ATCSs,  when  exploring  the  ATC  workstation  for  information,  have  a  set  of  beliefs 
about  the  current  situation,  break  that  situation  up  in  subsets,  and  execute  sequences  of  fixations 
to  verify  or  update  the  knowledge  about  the  current  situation.  It  is  our  belief  that  the  scanning  or 
monitoring  process  executes  in  parallel  with  these  processes.  Knowledge  of  the  resulting  output 
in  the  form  of  a  visual  scanning  pattern  may  not  tell  us  much  about  the  architecture  of  those 
cognitive  processes.  From  the  literature,  we  have  learned  that  “good  scanners"  will  show  little 
structure  in  the  visual  scan  in  an  attempt  to  cover  most  of  the  information  display,  but  we  cannot 
assume  that  ATCS  visual  scanning  solely  consists  of  a  monitoring  process  that  needs  to  detect 
conspicuous  events.  Research  in  other  areas  such  as  scanning  of  x-rays  by  radiologists  suggests 
that  shorter  fixations  relate  to  the  monitoring  process,  whereas  the  longer  “evaluation”  fixations 
directly  relate  to  decision  making  and  more  detailed  diagnosis  (Carmody,  Nodine,  &  Kundel, 
1981).  Eye  movements  in  the  ATCS  task  definitely  have  an  action-oriented  component  to  it,  as 
well.  Gross  (1998)  found,  for  example,  that  fixation  on  an  aircraft  that  was  about  to  be  subject  to 
a  communication  event  increased  in  duration  up  to  40  seconds  before  the  communication  event 
took  place.  During  the  communication  event  itself,  fixations  durations  were  not  longer  or  did  not 
take  place  on  the  aircraft.  This  finding  may  indicate  that  an  ATCS  absorbed  information  from 
the  aircraft  presentations,  planned  the  communication  action,  and  then  executed  the  action 
without  visually  obtaining  further  information  about  the  aircraft.  An  R-side  ATCS  will  have  the 
results  of  these  type  of  action-oriented  visual  scanning  outcomes  interspersed  with  the  outcomes 
related  to  situation  assessment  (i.e.,  event  detection).  Therefore,  the  summary  variables  reported 
in  previous  studies  did  not  distinguish  monitoring  from  ATC-related  activities  involved  in 
delivering  a  clearance.  The  eye  movements  of  the  D-side  ATCS  reported  here,  however,  have 
removed  most  of  the  monitoring  process  because  the  D-side  ATCS  has  a  very  different  function 
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than  the  R-side  ATCS.  The  resulting  eye  movements  may  very  well  depict  more  of  the  action 
oriented  eye  movement  behavior  than  we  have  previously  found  in  R-side  ATCSs. 

For  the  scene-based  eye-movement  characteristics,  the  fixation  and  saccade  results  indicate  that 
D-side  ATCSs  spent  a  great  amount  of  time  away  from  the  radar  display.  Currently,  D-side 
ATCSs  act  as  a  second  pair  of  eyes  for  the  R-side  ATCS;  however,  the  introduction  of 
automation  changed  this  and  drew  the  visual  scanning  of  the  D-side  to  the  automated  tool.  With 
the  introduction  of  automation,  the  assistance  the  D-side  can  offer  the  R-side  ATCS  changes. 
Thus,  automation  tools  have  substantial  implications  for  the  future  role  of  the  D-side. 

Descriptive  statistics  show  that  D-side  ATCSs  did  not  utilize  the  trial  plans  capability  of  the 
DST.  This  indicates  that,  most  likely,  the  ATCSs  identified  conflicts  from  the  DST.  However, 
they  conceptualized  their  own  solutions  to  those  conflicts  without  accessing  any  possible 
solutions  from  the  DST.  This  may  also  indicate  that  the  ATCSs  trusted  the  DST  to  predict  and 
identify  conflicts  but  not  to  provide  solutions  to  those  conflicts.  Further,  the  ATCSs  were  novice 
DST  users,  and  this  may  have  had  an  impact  on  using  the  trial  plans  capability  less. 

The  object-based  eye-movement  characteristics  showed  that  the  number  of  fixations  was  higher 
for  the  FDB,  although,  as  automation  increased,  the  number  of  fixations  on  the  FDBs  decreased. 
ATCSs  also  spent  more  time  fixating  on  the  FDB,  particularly  under  high  task  load  conditions. 

High  task  load  and  limited  and  full  automation  led  to  lower  levels  of  structure  in  the  ATCS’s 
visual  scan.  The  D-side  ATCS  seems  able  to  follow  events  better  under  low  task  load 
conditions.  Both  automation  conditions  most  likely  reduce  visual  scan  structure  because  the 
ATCS  becomes  increasingly  strategic  in  focus.  ATCSs’  plans  no  longer  come  from  the  layout  of 
the  airspace  but  from  the  information  provided  by  the  DST.  Under  no  automation,  the  radar 
display  provides  the  feedback  used  in  the  scan. 

In  Figure  46,  under  conditions  without  automation,  ATCSs  still  fixated  10-15  %  of  the  time  on 
the  display  on  the  D-side.  For  the  purposes  of  the  analyses,  we  did  not  remove  any  fixations  or 
saccades  on  the  SAVANT  queries  or  during  the  first  minute  (i.e.,  during  the  countdown),  which 
occurred  on  the  same  display  as  the  DST.  In  other  words,  there  were  fixations  and  saccades  on 
the  DST  even  when  the  DST  was  not  in  use;  these  fixations  and  saccades  are  reflected  by  the 
none  zero  values  in  the  no-automation  condition  and  would  account  for  approximately  250 
fixations  under  high  task  load  conditions.  The  remaining  fixations  on  the  D-side  display  under 
conditions  without  automation  are  most  likely  fixations  during  transitions  to  other  scene  planes 
such  as  the  D-side  keyboard,  the  maps,  and  the  flight  strip  bays.  The  number  of  these  fixations  is 
a  systematic  error,  that  is,  they  exist  during  all  automation  levels.  Comparisons  between 
automation  conditions,  therefore,  are  still  valid,  although  the  absolute  value  of  the  number  of 
fixations  is  lower  than  those  used  in  the  analyses. 

3.4  Communications 

During  normal  operations  within  an  ATC  team  (R-  and  D-side  ATCSs),  the  two  ATCSs  have 
predefined  responsibilities  as  defined  by  FAA  Order  71 10.65L  (Federal  Aviation  Administration, 
1998)  and  local  orders.  Each  member  of  the  ATC  team  has  specific  communication 


67 


responsibilities  (e.g.,  R-side  ATCS  communicates  with  pilots).  In  addition,  the  team  members 
develop  communication  patterns  between  them. 

Changes  in  the  way  an  ATCS  team  works  may  manifest  itself  in  intrateam  communications, 
ATCS  to  pilot  and  ATCS  to  ATCS  in  adjoining  sector  communications.  Although  considerable 
research  has  focused  on  the  effects  that  decision  aids  will  have  on  pilot-to-ATCS 
communication,  relatively  little  research  exists  on  how  DSTs  will  affect  ATCS  to  ATCS 
communication  (Kanki  &  Prinzo,  1996).  We  know  very  little  about  the  daily  task-related 
communication  exchanges  between  R-side  and  D-side  ATCSs.  This  type  of  communication 
becomes  especially  important  when  the  new  DSTs  attempt  to  enhance  the  tactical  and  strategic 
decision-making  capabilities  of  ATCSs.  What  effect  might  these  new  technologies  have  on& 
existing  patterns  of  intrateam  communications  and  communications  between  ATCSs  within 
adjoining  sectors?  Without  an  understanding  of  the  existing  patterns  of  intrateam 
communications,  there  is  no  empirical  way  to  answer  that  question. 

Because  the  recordings  required  by  the  NAS  do  not  capture  the  internal  communication  between 
ATC  team  members,  we  videotaped  all  task-related  R-side/D-side  communications.  An  ATC 
SME  coded  communications  using  the  C4T  (Peterson  et  ah,  in  press).  The  C4T  has  three 
communication  categories:  the  topic  of  communication,  the  grammatical  form  of 
communication,  and  the  method  of  communication.  Thus,  the  C4T  captures  the  “what”  (topic) 
and  “how”  (form  and  method)  of  communication. 

A  frequency  analysis  of  the  communications  between  the  D-side  and  R-side  resulted  in  the 
following  observations.  Most  of  the  intrateam  communications  were  aircraft  traffic  (40%)  and 
route  of  flight  (15%).  The  least  frequent  communications  were  inter-sector  coordination  (1%). 
R-side  and  D-side  ATCSs  did  not  statistically  differ  in  the  topic  of  their  communication. 
However,  they  did  differ  in  the  grammatical  form  of  their  communication.  Whereas  the  D-side 
had  a  higher  percentage  of  statements  and  observations  (56%  vs.  30%),  the  R-side  had  a  higher 
percentage  of  answers  (43%  vs.  35%).  It  appeared  that,  compared  to  the  R-side,  the  D-side 
ATCS  was  the  initiator  of  more  communication.  Finally,  there  was  no  intrateam  difference  in 
the  method  of  communication.  The  use  of  voice  alone  was  the  most  frequent  method  of 
communicating  (70%).  The  remaining  30%  of  communication  contained  a  mixture  of  verbal  and 
non-verbal  expression.  This  latter  finding  suggested  that  any  changes  affecting  the  line  of  sight 
between  R-side  and  D-side  positions  could  disrupt  the  adaptive  use  of  intrateam  nonverbal 
communication. 

One  of  the  more  consistent  findings  in  the  literature  on  ATCS-to-pilot  voice  communications  is 
that  workload  affects  the  quality  and  quantity  of  communication  exchanges  (Prinzo  &  Britton, 
1993).  In  this  literature,  workload  is  primarily  measured  by  the  number  of  aircraft  at  a  given 
time  (i.e.,  task  load)  that  are  under  the  control  of  an  R-side  ATCS.  As  task  load  increases,  there 
is  a  corresponding  trend  toward  an  increasing  number  of  communication  errors  (Morrison  & 
Wright,  1989).  Research  suggests  that  as  ATCSs  and  pilots  become  overburdened,  the  clarity  of 
their  communications  begins  to  suffer.  This,  in  turn,  puts  ATCSs  and  pilots  at  a  higher  risk  of 
committing  readback/hearback  errors  (Morrison  &  Wright).  Standard  ATC  protocol  requires 
pilots  and  ATCSs  to  repeat  what  they  heard  so  that  the  sender  knows  that  ATCSs  received 
messages  accurately.  A  readback/hearback  error  occurs  when  an  incorrect  pilot  or  ATCS 
readback  of  information  goes  uncorrected. 
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Changes  in  workload  also  affect  the  kinds  of  communication  exchanges  that  occur.  In  a  study  of 
ATC  communications  in  a  simulated  en  route  environment,  as  workload  increased,  ATCSs  in 
top-rated  teams  issued  more  communication  reports  to  pilots  than  did  lower-rated  teams. 
Compared  to  lower-rated  teams,  top-rated  teams  issued  shorter  messages  as  a  means  of  insuring 
accuracy  (Human  Technology  Incorporated,  1991).  ATC  SMEs  conducted  OTS  ratings  of  team 
performance.  However,  as  is  the  case  with  experiments,  whether  these  results  generalize  to  the 
broader  ATC  population  or  are  specific  to  a  given  experimental  manipulation  is  not  yet  clear. 

Although  related  to  workload,  the  effects  of  technology  on  communication  are  unclear.  From  the 
concept  of  monitoring,  an  operator  has  no  need  to  communicate  unless  some  event  occurs  that 
requires  the  actions  of  another.  Whether  an  operator  uses  or  trusts  technology,  one  might  argue 
that  communications  would  increase  during  the  transition  period  of  adjusting  to  the  new 
technology.  This  would  be  especially  true  if  there  were  problems  with  the  technology  and  team 
members  had  to  decide  whether  to  trust  it.  However,  once  the  transition  period  passes,  one 
would  expect  communications  to  return  to  a  previous  baseline. 

3.4.1  Intrateam  Communications 

Appendix  X  contains  descriptions  of  C4T  sub  levels  within  each  of  the  three  categories.  For 
information  on  the  development  and  operational  validation  of  the  C4T,  see  Peterson  et  al.  (in 
press).  The  intrateam  communications  analyses  represent  a  joint  project  between  the  RDHFL 
and  CAMI.  As  part  of  their  requirements  at  CAMI,  Bailey,  Willems,  and  Peterson  (in  review) 
wrote  a  paper  on  this  information. 

3 .4. 1 . 1  Data  Analysis  and  Results 

Because  the  communication  component  of  the  experiment  was  primarily  descriptive  in  nature, 
we  did  not  replace  missing  data  with  mean  substitutions  but,  instead,  dropped  the  cases  from 
further  analysis.  In  all,  we  dropped  three  teams  from  the  analyses.  Of  the  remaining  five  teams, 
we  coded  3,194  communication  events. 

To  determine  if  differences  existed  in  the  patterns  of  communication  operating  between  the  field 
study  and  the  experiment,  we  analyzed  the  frequency  data  by  topic,  format,  and  expression. 
Although  we  did  not  model  the  experiment  after  a  particular  en  route  ARTCC,  the  scenarios 
reflected  real  world  events.  Thus,  one  might  expect  similarities  between  intrateam 
communications  within  a  field  and  experimental  setting. 

Table  8  shows  comparisons  for  the  percentage  of  R-side  and  D-side  communications  related  to 
the  topic  of  communication,  its  grammatical  form,  and  the  method  of  expression.  Although  the 
percentages  differed  for  the  topics  of  communication  presented  in  Table  8,  both  lab  and  field 
assessments  identified  the  same  top  three  topics.  These  included  communications  about  traffic, 
route  of  flight,  and  aircraft  altitude.  Compared  to  the  field,  the  most  noticeable  difference  in  the 
experiment  was  the  lack  of  communications  about  the  weather,  point-outs,  and  traffic  flow. 

There  was  also  little  communication  about  flight  strips. 

As  Table  8  shows,  the  grammatical  form  of  communications  also  differed  between  the  two 
environments.  The  field  results  show  a  strong  tendency  for  the  D-side  to  make  statements 
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(55.9%)  and  the  R-side  (42.8%)  to  provide  answers.  In  contrast,  for  the  experiment,  the  results 
show  both  R-side  and  D-side  predominately  making  statements  (58%  vs.  77.3%,  respectively). 

Verbal  communication  is  the  method  of  choice  for  R-side  and  D-side  ATCSs.  However,  as 
Table  8  shows,  in  the  experiment,  the  D-side  had  a  stronger  tendency  to  use  a  mixture  of  verbal 
and  nonverbal  expressions  than  did  the  R-side  (24.7%  vs.  5%).  For  the  field  setting,  the 
percentage  of  mixed  messages  was  similar  for  both  the  R-side  and  D-side  (14.6%  vs.  16.8%). 
The  two  settings  also  differed  in  the  percentage  of  nonverbal  communications  used.  In  the  field, 
1 3.9%  of  the  communications  was  solely  nonverbal  for  both  the  R-side  and  D-side.  This  is  in 
contrast  to  the  lower  percentages  recorded  during  the  experiment  (R-side  0.5%,  D-side  2.8%). 


Table  8.  Comparisons  of  R-side  and  D-side  communications  in  field  and  laboratory  setting 


En  route  Center 

Laboratory  Setting 

R-side% 

D-side% 

R-side% 

D-side% 

Communication  Topic 

Traffic 

41.0 

37.9 

53.7 

51.2  | 

Route  of  flight 

14.2 

15.6 

13.1 

11.7 

Altitude 

7.1 

8.0 

16.0 

21.1 

Weather 

5.5 

6.8 

0.0 

0.0 

Point-out 

5.0 

6.1 

0.0 

0.0 

Traffic  flow 

5.2 

5.6 

0.0 

0.0 

Frequency 

5.9 

4.7 

3.5 

2.7 

Flight  Strips 

5.6 

4.5 

0.7 

1.0 

Equipment 

3.3 

4.0 

4.5 

4.9 

Hand-off 

3.6 

3.1 

2.9 

1.8 

Speed 

2.6 

2.8 

4.4 

2.8 

Approval 

1.0 

0.9 

1.0 

1.8 

Communication  Format 

Statement 

29.7 

55.9 

58.0 

77.3 

Answer 

42.8 

25.1 

18.3 

10.4 

Question 

12.2 

16.4 

22.9 

11.3 

Command  Answer 

5.8 

0.3 

0.0 

0.0 

Command 

0.5 

2.4 

0.8 

1.0 

Communication  Expression 

Verbal 

77.1 

69.3 

93.9 

69.0 

Verbal  &  Nonverbal 

14.7 

16.8 

5.0 

24.7 

Nonverbal 

13.9 

13.9 

0.5 

2.8 

Equipment 

0.0 

0.1 

Equipment  &  Verbal 

0.6 

3.4 

Equipment  &  Nonverbal 

0.0 

0.1 

We  conducted  a  2  x  3  (task  load  x  automation)  within-subjects  ANOVA  for  the  composite 
number  of  communication  exchanges  of  team  members  and  did  not  find  any  statistically 
significant  results.  A  power  analysis  revealed  that  we  had  insufficient  power,  but  we  present  the 
data  and  trends  to  add  to  the  knowledge  base  of  this  poorly  studied  area.  When  we  analyzed 
each  of  the  three  C4T  categories  separately,  we  found  secondary  trends  for  task  load  on  general 
communications  about  a  specific  aircraft  and  communications  involving  altitude  changes  (Figure 
63  and  Figure  64).  More  communications  occurred  for  specific  aircraft  and  altitude  changes 
under  high  task  load  conditions.  In  addition,  there  was  a  secondary  tendency  for  more 
information  to  be  exchanged  on  specific  aircraft,  altitude,  and  route  of  flight  under  low  task  load, 
full  automation  (secondary  trend;  Figure  65  and  Figure  66). 
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Figure  63.  Number  of  specific  aircraft  communications  by  task  load. 
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Figure  64.  Number  of  altitude  communications  by  task  load. 
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Figure  65.  Number  of  specific  aircraft,  altitude,  and  route  communications  by  low  task  load  and 
automation. 
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Figure  66.  Number  of  specific  aircraft,  altitude,  and  route  communications  by  high  task  load  and 
automation. 
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3. 4. 1.2  Discussion 


The  C4T  profile  of  the  laboratory  experiments  (collapsing  across  all  conditions)  compared 
favorably  with  data  collected  from  the  field.  In  both  cases,  the  top  three  topics  of 
communication  were  about  traffic,  route  of  flight,  and  aircraft  altitude.  Also,  in  both  cases,  voice 
only  was  the  dominant  mode  of  communications.  Despite  these  similarities,  there  were  notable 
differences  in  the  grammatical  form  of  communication  between  R-side  and  D-side  ATCSs.  In 
the  field,  but  it  appeared  that  the  R-side  primarily  responded  to  statements  by  the  D-side,  in  the 
experiment,  it  appeared  that  both  R-side  and  D-side  ATCSs  were  issuing  statements.  More 
nonverbal  communications  occurred  in  the  field  as  compared  to  the  laboratory  environment. 

This  may  be  because,  in  the  field,  team  members  are  familiar  with  each  other,  but  in  the 
laboratory,  they  did  not  know  each  other  before  participating. 

Although  only  a  secondary  trend,  more  communication  exchanges  occurred  in  the  high  task  load 
conditions  as  compared  to  the  low  task  load  conditions.  This  was  especially  true  concerning 
communications  about  a  specific  aircraft  and  about  altitude  changes.  The  effects  of  automation 
on  communications  were  not  as  clear.  There  was  some  evidence  that  in  the  low  task  load 
conditions  more  communication  exchanges  occurred  for  specific  aircraft,  altitude,  or  route  of 
flight  using  the  full  automation  as  compared  to  limited  or  no  automation. 

3.4.2  Push-to-Talk 


We  collected  LL  (ATCS  to  ATCS  in  adjoining  sectors)  and  (R-side  ATCS  to  simulation  pilots) 
communications.  For  the  purposes  of  analysis,  these  communications  apply  to  the  team  of 
ATCSs  instead  of  each  ATCS  individually. 

3 .4.2. 1  Data  Analysis  and  Results 

We  created  one  communications-related  data  set  containing  LL  and  Radar-side  (R) 
communications.  To  enable  measures  of  SA  with  SAGAT,  scenarios  had  different  durations.  To 
eliminate  the  effect  of  different  scenario  lengths,  we  corrected  for  duration  and  expressed  the 
number  of  communications  in  events  per  minute.  We  averaged  the  number  and  duration  of  each 
communication  type,  per  scenario  and  team  of  ATCSs,  and  performed  analyses  on  the  eight 
teams.  We  conducted  2x3  (task  load  x  automation)  within-subjects  MANOVAs  for  the  LL  and 
R  communication  measures,  respectively.  We  followed  any  significant  MANOVA  results  with 
subsequent  ANOVAs  and  used  an  adjusted  alpha  of  .025.  Appendix  Y  contains  the  means,  SDs, 
MANOVA,  and  ANOVA  results. 

3.4.2. 1 . 1  Mean  Number/Duration  of  PTT  Landline  Communications 

The  within-subjects  MANOVA  showed  significant  effects  for  task  load,  automation,  and  the  task 
load  x  automation  interaction  [A  =  .29,  F( 2,6)  =  7.22,  p  <  .05;  A  =  .1 1,  F(4,4)  =  .80,  p  <  .05;  A  = 
.13,  F(4,4)  =  6.96,  p  <  .05,  respectively,  Appendix  Y,  Table  Y-3].  Consequently,  we  performed 
univariate  analyses  on  the  number  and  duration  of  LL  communications. 

The  ANOVA  for  the  mean  number  of  LL  communications  showed  a  significant  main  effect  for 
task  load  [F(l,7)  =  16.1 8,p  <  .025,  Table  Y-4],  ATCSs’  average  number  of  communications 
increased  as  the  task  load  increased  (Figure  67). 
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Figure  67.  Mean  number  of  LL  communications  per  minute  by  task  load. 

The  ANOVA  for  the  mean  duration  of  LL  communications  showed  a  significant  task  load  x 
automation  interaction  [F(2,14)  =  4.93,  p  <  .025,  Table  Y-5],  The  simple  effect  of  task  load 
within  full  automation  was  significant  [F(l,7)  =  1 1.03,  p  <  .025,  Table  Y-5],  ATCSs’  mean 
duration  of  communications  was  longest  during  the  full  automation,  low  task  load  scenario 
(Figure  68). 


Low  Task  Load  -  High  Task  Load 


Figure  68.  Mean  duration  of  LL  communications  by  task  load  and  automation. 

3 .4.2.1 .2  Mean  Number/Duration  of  PTT  R-side  Communications 

The  2x3  (task  load  x  automation)  within-subjects  MANOYA  examining  R  communications 
showed  a  significant  effect  for  task  load  [A  -  .023,  F( 2,6)  =  130.21,/?  <  .0001,  Table  Y-6]. 
Because  we  found  significant  effects  from  the  MANOVA,  we  performed  subsequent  ANOVAs. 

The  ANOVA  for  the  mean  number  of  R  communications  showed  a  significant  main  effect  for 
task  load  [F(l,7)  =  148.07 ,p  <  .05,  Table  Y-7].  ATCSs’  mean  number  of  communications 
increased  as  task  load  increased  (Figure  69). 
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Figure  69.  Mean  number  of  R  communications  per  minute  by  task  load. 

The  ANOVA  for  the  mean  duration  of  R  communications  showed  a  significant  main  effect  for 
task  load  [F(l,7)  =  1 1.73,  p  <  .05,  Table  Y-8].  ATCSs  mean  duration  of  communications 
increased  as  task  load  decreased  (Figure  70). 
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Figure  70.  Mean  duration  of  R  communications  by  task  load. 
3.4.2.2  Discussion 


Low  Task  Load  High  Task  Load 


Although  not  statistically  significant,  a  trend  revealed  the  duration  of  communications  was 
longest  when  in  the  fully  automated  condition.  Further,  an  interaction  between  task  load  and 
automation  showed  that  task  load  significantly  influenced  the  duration  of  communications  in  the 
full-automation  condition.  ATC  teams’  LL  communications  lasted  longer  when  task  load  was 
low  and  full  automation  was  present.  This  would  seem  to  indicate  that  when  task  load  was  low 
and  ATCSs  had  full  use  of  the  DST,  they  had  more  detailed  ideas  of  how  to  fix  potential 
conflicts  and  discussed  these  solutions  longer  with  the  controlling  sector  of  the  aircraft.  The 
ATC  teams’  use  of  the  LL  increased  with  increasing  task  load.  The  higher  traffic  volume 
associated  with  high  task  load  presented  greater  numbers  of  potential  conflicts  that  ATCSs 
needed  to  resolve. 

Likewise,  for  the  R  communications,  the  number  of  communications  increased  as  task  load 
increased.  The  R  communications  accounted  for  the  communications  between  the  R-side 
ATCSs  and  the  simulation  pilots.  As  the  amount  of  traffic  increased,  we  expected  that  R-side 
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ATCSs  would  increase  the  number  of  communications  they  initiated.  The  higher  the  volume  of 
traffic,  the  more  actions  and  clearances  the  R-side  ATCS  needs  to  communicate.  In  addition,  the 
duration  of  the  communications  decreased  when  traffic  task  load  was  high.  Once  again,  the  R- 
side  ATCS  needed  to  issue  a  larger  number  of  clearances  and  decreased  the  duration  of  these 
communications  in  order  to  get  them  all  in.  This  corresponds  to  the  Human  Technology 
Incorporated  (1991)  findings  that  ATCSs  in  top-rated  teams  compensated  for  increases  in 
workload  by  increasing  communications  while  decreasing  their  durations.  Willems  and  Truitt 
(1999)  also  found  that  the  number  of  communications  increased  as  task  load  increased;  however, 
they  did  not  find  a  change  in  duration  nor  did  they  examine  LL  communications.  In  the  current 
study,  even  our  low  task  load  scenarios  had  higher  volumes  of  traffic  than  what  the  ATCSs 
experienced  in  the  Willems  and  Truitt  study  because  our  sector  staffing  included  an  R-  and  D- 
side,  where  Willems’  and  Truitt’s  study  staffed  the  sector  with  a  single  R-side  only.  The  more 
traffic,  the  more  ATCSs  need  to  compensate  by  their  communications.  Under  increasing  task 
loads,  they  need  to  issue  more  clearances  but  have  the  same  amount  of  time  to  get  them  in, 
therefore  shortening  the  amount  of  time  they  communicate. 

3.5  Trust  Ratings 

Several  researchers  have  proposed  taxonomies  to  describe  the  dimensions  of  trust.  Barber 
(1983)  focused  on  the  expectations  a  user  brings  to  a  trusting  relationship.  When  experience 
confirms  each  expectation,  the  development  of  an  accurate  mental  model  of  the  system  is  more 
likely.  Rempel,  Holmes,  and  Zanna  (1985)  focused  on  the  dynamics  of  trust,  which  evolve  and 
change  over  time  because  of  experience  with  a  system.  Zuboff  (1988)  and  Lee  and  Moray 
(1992)  synthesized  the  taxonomies  of  Barber  and  Rempel  et  al.  and  proposed  their  own 
taxonomies  of  trust.  Moffa  (1994)  measured  user  trust  with  a  questionnaire  he  constructed  from 
Barber’s  and  Rempel  et  al.’s  trust  metrics.  He  varied  level  of  agreement  between  the  user’s 
diagnosis  and  that  offered  by  a  medical  diagnostic  aid.  He  varied  experience  with  the  system  in 
the  form  of  number  of  trials.  Moffa  found  that  trust  decreased 

•  with  consistent  system  disagreement  for  both  levels  of  trials; 

•  with  consistent  system  disagreement  but  increased  with  honest  agreement,  when  pooled 
across  trials;  and, 

•  with  more  experience  with  the  system,  when  pooled  across  level  of  agreement. 

Moffa  (1994)  found  no  significant  difference  between  ratings  for  Barber’s  (1983)  dimensions 
and  those  for  Rempel  et  al.’s  (1985)  dimensions.  Whereas,  Moffa  studied  trust  and  behavior  of 
non-experts  responding  to  discrete,  case-based  automation  in  medicine,  Moffa  and  Stokes  (1997) 
examined  this  work  in  light  of  other  domains,  including  ATC.  An  ATCS  works  in  a  domain  that 
is  more  continuous  and  process-based.  Moffa  and  Stokes  concluded  that  it  is  likely  that  the 
structure  of  knowledge  in  a  domain,  the  workplace  practices,  and  the  professionalism  of  the 
domain  affect  trust,  complacency,  and  compensatory  behaviors. 
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We  applied  work  done  in  operator  trust  in  automation  to  the  domain  of  ATC.  Based  on  Barber 
(1983),  Rempel  et  al.  (1985),  Zuboff  (1988),  and  Lee  and  Moray  (1992),  we  propose  the 
following  taxonomy  to  describe  the  dimensions  of  trust  in  ATC.  Operator  experience  forms  the 
basis  of  this  mostly  linear  taxonomy. 

•  We  assume  that  users  of  air  traffic  systems  enter  the  human-automation  relationship  with  the 
following  expectations  (from  Barber): 

_  The  system  is  appropriate  for  the  task  for  which  it  was  designed,  and  it  will  remain  so. 

_  The  system  exhibits  technical  competence. 

_  The  system  is  responsible. 

•  Each  user  explores  a  new  system  by  “trying  out”  its  functions  and  features,  each  in  turn. 
Exploration  continues  when  the  user  applies  the  system  to  each  new  situation. 

•  With  experience,  the  user  sees  how  consistent  (or  inconsistent)  the  system’s  recurrent 
behaviors  are  (from  Rempel  et  al.). 

•  With  continued  experience,  the  user  shifts  from  evaluating  specific  behaviors  to  evaluating 
the  dependability  of  the  entire  system  (from  Rempel  et  al.). 

•  Given  enough  experience  with  a  predictable  and  dependable  system  that  is  appropriate, 
competent,  and  responsible,  the  user  will  have  faith  that  the  system  will  continue  to  be  so. 

3.5.1  Data  Analysis  and  Results 


Four  items,  rated  on  a  10-point  scale,  comprised  the  trust  scale  for  the  DST:  the  predictability  of 
the  DST’s  behavior,  how  well  the  DST  predicted  the  outcome  of  separation  strategies,  the 
capability  of  the  DST  to  predict  future  conflicts,  and  the  technical  knowledge  incorporated  in  the 
tool.  Because  participants  only  completed  these  items  in  either  the  limited-  or  full-  automation 
condition,  we  conducted  a  2  x  2  x  2  (position  x  task  load  x  automation)  within-subjects 
MANOVA.  Appendix  Z  contains  the  means,  SDs,  MANOVA  and  ANOVA  tables. 

At  the  multivariate  level,  a  significant  effect  for  automation  occurred  [A  =  .38,  F(4,12)  =  4.97, 
p  <  .05,  Appendix  Z,  Table  Z-2].  Because  the  MANOVA  results  indicated  a  significant  effect 
for  automation,  we  conducted  ANOVAs  on  each  of  the  measures  and  adjusted  the  alpha  to  .013 
to  control  for  error.  At  the  univariate  level,  automation  did  not  reach  significance;  however,  we 
found  several  trends  in  the  data  using  an  alpha  of  .05.  ATCSs  rated  the  DST  as  having  better 
prediction  capabilities  in  the  full  automation  condition  than  in  the  limited  automation  condition 
(Figure  71).  ATCSs  perceived  the  DST  to  behave  more  predictably  under  low  task  load 
conditions  than  high  task  load  conditions,  although  the  degree  of  automation  affected  this.  For 
the  limited  automation  condition,  task  load  did  not  have  an  effect;  however,  in  the  full 
automation  condition,  ATCSs  perceived  the  DST  as  more  predictable  when  the  task  load  was 
low  (Figure  72).  Finally,  ATCSs  rated  the  DST  as  more  capable  to  predict  future  conflicts  when 
the  traffic  task  load  was  low  (Figure  73). 
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Figure  71 .  How  well  the  DST  predicted  the  outcome  by  automation. 
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Figure  72.  The  predictability  of  the  DST’s  behavior  by  task  load  and  automation. 
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Figure  73.  Capability  of  the  DST  to  predict  future  conflicts  by  task  load. 
3.5.2  Discussion 


Task  load  and  automation  affected  ATCS  trust  in  the  DST.  The  algorithms  that  determined  the 
DST’s  behavior  obviously  did  not  change  between  conditions.  Therefore,  ATCSs  trust  in  the 
DST  changed.  It  is  quite  likely  that  the  lack  of  a  mental  model  for  the  DST  made  it  difficult  for 
ATCSs  to  understand  how  the  DST  arrived  at  certain  conclusions.  During  training,  we  witnessed 
the  ATCSs’  disbelief  that  a  tool  such  as  the  DST  existed  and  that  it  could  predict  potential 
conflicts  between  aircraft.  Although  training  may  have  modified  the  ATCSs’  expectations  and 


77 


beliefs  regarding  the  DST’s  prediction  capabilities,  it  is  most  likely  that  the  limited  training  and 
period  of  use  of  the  DST  did  not  completely  modify  their  expectations  nor  give  them  a  chance  to 
form  a  complete  mental  model  of  how  the  DST  worked. 

The  graphical  display  and  trial  planning  was  only  available  under  full  automation.  Therefore,  the 
reduction  in  trust  under  limited  automation  may  be  related  to  the  lack  of  these  features.  In  the 
limited  automation  condition,  only  the  aircraft  list  was  available.  In  the  aircraft  list,  potential 
conflicts  are  indicated  by  either  red  or  yellow  emphasis.  It  fails  to  indicate  which  aircraft 
conflict  with  one  another.  ATCSs,  therefore,  did  not  receive  the  feedback  they  needed  to  verify 
the  conflicting  aircraft  unless  they  accessed  the  relevant  information  through  the  graphical 
display  (see  Figure  3).  This,  in  turn,  may  result  in  a  lower  level  of  trust  in  the  DST.  As  noted  in 
the  trust  taxonomy,  the  expectations  are  that  the  system  is  appropriate  for  the  task,  and,  in  this 
case,  limited  automation  is  not  entirely  appropriate  for  the  task.  These  types  of  issues  are  the 
focus  of  the  human  computer  interface  design.  By  providing  ATCSs  the  ability  to  group  aircraft 
when  they  belong  to  the  same  potential  conflict,  we  can  give  them  a  better  feel  for  the 
correctness  of  the  information.  This  seems  especially  relevant  in  operations  where  a  single  R- 
side  ATCS  works  a  sector  and  often  only  displays  the  aircraft  list. 

Under  high  task  load  conditions  (i.e.,  when  ATCSs  needed  assistance  most),  they  trusted  the 
DST  the  least.  During  training  and  implementation,  this  may  be  an  issue  to  give  more  emphasis. 
Further,  as  noted  by  Wiener  (1988),  users  of  automation  may  turn  it  off  under  high  workloads. 

In  the  current  study,  ATCSs  did  not  have  the  ability  to  turn  the  automation  off  but  could 
disregard  the  information  it  presented.  This  could  lead  the  ATCSs  to  believe  the  automation  was 
not  as  predictable  when  task  load  was  high  and  full  automation  was  in  use. 

The  results  imply  that  both  task  load  and  level  of  automation  affected  ATCSs’  trust  in  the  DST 
in  this  study.  ATCSs  described  better  prediction  quality  for  outcome  of  separation  strategies 
when  able  to  use  all  the  features  offered  by  the  DST  such  as  trial  flight  plans  and  the  graphic 
plans  display,  relating  to  the  appropriateness  of  the  tool  for  the  given  task.  Low  task  load 
scenarios  allowed  ATCSs  more  time  to  work  with  the  DST  and  explore  the  various  functions  of 
the  DST.  This  coincides  to  the  trust  taxonomy  described  previously.  With  increased  amounts  of 
time  to  “try  out”  the  DST,  the  ATCSs  viewed  it  more  favorably. 

3.6  Real-Time  Objective  Performance 

To  obtain  the  Real-Time  Objective  Performance  variables  that  the  Data  Reduction  and  Analysis 
(DRA)  system  at  the  RDHFL  previously  provided,  we  submitted  the  TGF  recordings  to  the  Data 
Reduction  and  Analysis  Tool  (DRAT). 

3.6.1  Data  Reduction  Analysis  Tool 

The  automated  data  reduction  module  developed  for  the  TGF  data  provided  performance  data  for 
complexity  and  handoff  efficiency,  conflicts,  and  task  load.  We  calculated  the  variables  based 
on  ATCS  responsibility,  not  on  fully  active  control  (i.e.,  data  block  maintenance  and 
communications  active).  In  other  words,  the  ATCS  was  responsible  for  any  aircraft  within  his  or 
her  sector  that  they  were  talking  to  or  that  had  already  changed  to  the  next  sector’s  frequency  but 
had  not  physically  reached  the  sector  boundary. 
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3 . 6 . 1 . 1  Data  Analysis  and  Results 


The  DRAT  module  processed  raw  data  files  produced  by  TGF.  DRAT  provided  summary, 
interval,  and  error  files  for  each  scenario.  The  interval  and  summary  files  comprised  two  data 
sets  in  spreadsheet  format.  The  first  data  set  contained  12x8  (scenarios  x  ATCS  teams)  records 
that  included  the  summary  variables  calculated  per  scenario.  The  second  data  set  contained  12  x 
10x8  (scenarios  x  intervals  x  ATCS  teams)  records  containing  the  summary  variables  calculated 
per  3-minute  interval.  For  the  first  10  participants,  we  averaged  any  replicated  data  for  a  team  of 
ATCSs. 

The  categories  of  variables  related  to  performance  included  Complexity  and  Flandoff  Efficiency 
and  Conflicts.  A  set  of  ANOVAs  tested  the  hypotheses  related  to  performance  scores  on 
selected  performance  variables.  These  ANOVAs  addressed  the  differences  across  scenarios  and 
were  of  a  repeated  measures  2x3  (task  load  x  automation)  design. 

3 .6. 1 . 1 . 1  Complexity  and  Handoff  Efficiency  Per  Scenario 

The  number  of  altitude,  heading,  and  speed  changes  comprised  the  complexity  items,  and  the 
number  of  handoffs  accepted  and  initiated  comprised  the  handoff  efficiency  items  that  we 
derived  from  DRAT.  We  calculated  the  number  of  altitude,  heading,  and  speed  changes  and  the 
number  of  handoffs  accepted  and  initiated  per  aircraft  to  adjust  for  the  varying  scenario  lengths 
that  ranged  in  duration  from  33  to  37  minutes.  We  performed  separate  2x3  (task  load  x 
automation)  within-subjects  ANOVAs  because  there  were  not  enough  degrees  of  freedom  to 
conduct  a  MANOVA.  The  adjusted  alpha  level  was  .01.  See  Appendix  AA  for  means,  SDs,  and 
ANOVA  tables. 

The  ANOVA  examining  the  number  of  speed  changes  showed  a  significant  effect  for  task  load 
[F(l,7)  =  19.79, p  <  .01,  Appendix  AA,  Table  AA-8].  Simulation  pilots  performed  more  speed 
changes  when  task  load  was  high  (Figure  74).  The  ANOVA  for  the  number  of  altitude  changes 
did  not  show  any  significant  effects,  although  a  trend  indicated  that  more  altitude  changes 
occurred  in  the  low  task  load  scenarios  than  in  the  high  task  load  scenarios  (Figure  75).  Finally, 
the  ANOVAs  for  the  number  of  heading  changes  and  the  number  of  handoffs  initiated  and 
accepted  did  not  show  any  significant  effects  for  the  experimental  variables. 
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Figure  74.  The  number  of  speed  changes  per  aircraft  by  task  load. 
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Figure  75.  The  number  of  altitude  changes  per  aircraft  by  task  load. 


We  conducted  separate  2x3x10  (task  load  x  automation  x  time  interval)  within-subjects 
ANOVAs  for  the  number  of  altitude,  heading,  and  speed  changes  per  3 -minute  interval.  We 
conducted  separate  2x3x9  (task  load  x  automation  x  interval)  within-subjects  ANOVAs  for  the 
number  of  handoffs  initiated  and  accepted  per  3-minute  interval.  We  did  not  include  the  first 
interval  in  these  analyses  because  it  included  the  start-up  of  the  scenario  where  the  system 
automatically  accepted  aircraft. 


The  ANOVA  for  the  number  of  altitude  changes  per  3-minute  interval  showed  significant  effects 
for  task  load,  interval,  and  the  task  load  x  interval  interaction  [F(9,63)  =  12.81,/?  <  .001,  F(  1,7)  = 
36.36,/?  <  .001,  F(9, 63)  =  4.04,/?  <  .001,  respectively,  Figure  76,  Table  AA-1 1],  More  aircraft 
from  the  high  task  load  condition  changed  their  altitude  than  aircraft  in  the  low  task  load 
condition.  Significantly  more  aircraft  had  altitude  changes  in  the  first  3-minute  interval  and 
gradually,  the  number  of  altitude  changes  decreased.  The  simple  effects  of  task  load  within  each 
interval  showed  significant  increases  in  altitude  changes  for  the  third,  seventh,  eighth,  and  tenth 
intervals  [F(l,7)  =  34.21,  14.59,  30.64,  31.03  all ps  <  .01,  respectively.  Table  AA-1 1]. 


Figure  76.  Number  of  altitude  changes  by  task  load  and  time  interval. 
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The  ANOVA  for  the  number  of  heading  changes  issued  showed  significant  effects  for  task  load 
and  interval  [F(l,7)  =  17.32  and  F(9,6 3)  =  6.83, ps  <  .01,  respectively,  Figure  77,  Table  AA-13]. 
More  heading  changes  occurred  in  the  high  task  load  scenario.  We  conducted  Tukey  HSD  post 
hoc  tests  to  determine  significant  differences  among  the  time  interval  means  (Table  AA-15). 
Several  means  were  significantly  different;  therefore,  we  provide  a  general  description.  As  the 
scenario  progressed,  the  number  of  heading  changes  generally  increased  reaching  a  peak  at  the 
ninth  interval  and  then  declining  at  the  tenth  interval. 
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Figure  77.  Number  of  heading  changes  by  task  load  and  time  interval. 

The  ANOVA  examining  the  number  of  speed  changes  showed  a  main  effect  for  task  load  and 
time  interval  [F(l,7)  =  60.62, p  <  .001,  F( 9,63)  =  7.54,/?  <  .0001,  respectively.  Figure  78,  Table 
AA-15].  The  number  of  speed  changes  increased  as  task  load  increased.  Tukey  HSD  post  hoc 
tests  showed  several  significant  differences  among  the  time  interval  means  (Table  AA-16).  In 
general,  as  the  scenario  increased,  the  number  of  speed  changes  increased  reaching  a  peak  at  the 
sixth  and  seventh  intervals  and  then  declining. 


3 

2.5 

2 


Figure  78.  Number  of  speed  changes  by  task  load  and  time  interval. 

Task  load  and  time  interval  influenced  the  number  of  handoffs  accepted  [F(l,7)  =  63.24,/?  < 
.001  and  F(8,56)  =  6.71,/?  <  .001,  respectively,  Figure  79,  Table  AA-17],  and  the  interaction 
between  task  load  and  time  interval  was  significant  [F(8,56)  =  7.49,/?  <  .001,  Table  AA-17]. 
ATCSs  accepted  more  handoffs  when  task  load  was  high.  Tukey  HSD  post  hoc  tests  showed 
several  significant  differences  among  time  interval  means  (Table  AA-1 8).  In  general,  ATCSs 
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Figure  79.  Number  of  handoffs  accepted  by  task  load  and  time  interval. 

Task  load  and  time  interval  affected  the  number  of  handoffs  initiated  [F(\,l)  =  93.52  and  F(8,56) 
=  8.91, /?s  <  .001,  respectively,  Figure  80,  Table  AA-19].  ATCSs  initiated  more  handoffs  when 
working  traffic  from  the  high  task  load  scenarios.  Tukey  HSD  post  hoc  tests  showed  significant 
differences  among  several  time  interval  means  (Table  AA-20).  In  general,  the  second  time 
interval  had  the  lowest  number  of  handoffs  initiated. 
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Figure  80.  Number  of  handoffs  initiated  by  task  load  and  time  interval. 

3 .6. 1 . 1 .3  Number  and  Duration  of  Conflicts 

Thirty  losses  of  separation  occurred  during  the  scenario  runs.  Due  to  the  relatively  low  number 
of  losses  of  separation,  we  did  not  conduct  ANOVAs  on  the  data.  Instead,  we  provide 
descriptive  statistics  on  the  data  (Appendix  AA).  The  distribution  for  the  mean  number  and 
duration  of  the  losses  of  separation  is  shown  in  Figure  81  and  Figure  82,  respectively.  More 
losses  of  separation  occurred  under  higher  task  loads  [M=  0.45  (0.63)]  and  with  either  limited 
[M=  0.50  (0.90)]  or  full  automation  [. M=  0.53  (0.52)].  The  mean  duration  of  losses  of 
separation  was  higher  under  high  task  load  and  no  automation  [M  =  13.3  (31.25)]. 
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Figure  8 1 .  Number  of  conflicts  by  task  load  and  automation. 
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Figure  82.  Duration  of  conflicts  by  task  load  and  automation. 


To  prevent  ATCSs  from  anticipating  when  scenarios  would  end,  we  stopped  simulations  at  times 
ranging  from  33  to  37  minutes.  To  enable  comparison  across  experimental  conditions,  we 
accounted  for  the  variable  scenario  lengths  by  calculating  the  number  of  aircraft  handled  per 
minute.  We  performed  a  2  x  2  x  3  (position  x  task  load  x  automation)  within-subjects  ANOVA. 
We  found  a  main  effect  of  task  load  [F(l,15)  =  270.95,  p  <  .001,  Figure  83,  Table  AA-23]. 
ATCSs  controlled  33%  more  aircraft  in  the  high  task  load  conditions. 


Figure  83.  Number  of  aircraft  handled  per  minute  during  scenario  by  task  load. 
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3.6.1.2  Discussion 


In  this  experiment,  we  observed  a  complex  interaction  of  the  effect  of  the  use  of  automation  and 
the  assistance  provided  to  the  R-side  ATCS  on  conflicts.  Although  the  use  of  the  DST  can 
benefit  the  R-side  ATCS  strategically,  it  will  remove  some  of  the  immediate,  more  tactical 
support.  Our  descriptive  analysis  of  the  instances  when  ATCSs  lost  separation  showed  that 
under  full  automation,  these  instances  were  more  frequent,  but  shorter.  Without  automation,  loss 
of  separation  occurred  less  frequently,  but  lasted  longer.  We  explain  this  from  the  D-side  ATCS 
activities  we  observed  under  these  conditions.  Without  automation  present,  the  D-side  ATCS 
physically  moved  closer  to  the  R-side  ATCS  and  provided  radar  assistance.  With  the  extra 
assistance,  the  ATCS  team  was  able  to  control  traffic  well  in  the  current  tactical  fashion.  The 
teams  resolved  tactical  or  immediate  problems  easily,  resulting  in  fewer  losses  of  separation. 
Because  the  team  now  worked  traffic  in  a  tactical  fashion,  however,  those  instances  that  did 
result  in  a  loss  of  separation  took  longer  to  resolve.  Under  full  automation,  the  D-side  ATCS  had 
the  DST  available  to  act  more  strategically  and  may  have  succeeded.  With  more  focus  on  the 
DST,  however,  the  R-side  ATCS  had  less  assistance  in  the  tactical  environment,  which  may  have 
resulted  in  more  tactical  problems  that  the  ATCS  could  resolve  quickly.  Another  possible 
explanation  for  this  effect  relates  to  earlier  findings  that  the  ATCSs’  trust  in  the  DST  was  low, 
particularly  in  high  task  load  conditions.  This  may  have  translated  into  more  losses  of  separation 
under  automation  because  the  D-side  ATCS  was  testing  the  correctness  of  the  DST;  he  or  she 
waited  to  see  if  a  predicted  conflict  actually  became  a  conflict. 

Although  the  number  of  times  ATCSs  did  not  maintain  adequate  separation  between  aircraft  was 
high  when  compared  to  operational  data,  the  data  cannot  be  used  for  formal  inferential  statistical 
analysis.  Even  under  high  traffic  conditions,  only  one  out  of  two  ATCSs  had  a  loss  of  separation 
when  full  automation  was  present.  Besides  the  fact  that  our  data  would  mostly  consist  of 
observations  that  equal  zero  cases  of  loss  of  separation,  our  data  do  not  indicate  if  a  loss  of 
separation  would  indeed  have  resulted  in  an  OE.  The  loss  of  separation  could  have  been  the 
result  of  a  pilot  deviation  or  could  have  been  rejected  as  an  OE  for  other  reasons. 

Our  data  seem  to  indicate  that  under  our  experimental  conditions,  the  high  traffic  task  loads  used 
in  our  scenarios  led  to  an  increased  frequency  of  loss  of  separation.  To  fully  understand  the 
sequence  of  events  that  led  to  a  loss  of  separation,  we  need  to  follow  a  different  approach  than 
we  have  followed  in  this  study.  We  have  data  that  we  synchronized  across  data  sets.  Therefore, 
we  can  trace  events  that  affected  aircraft  involved  in  a  conflict  in  an  attempt  to  explain  the  cause 
of  the  loss  of  separation.  Tracing  these  events  is  labor  intensive  and  tedious  but  may  provide 
useful  data  that  can  help  us  understand  how  ATCSs  loose  separation  between  aircraft. 

To  allow  other  researchers  to  compare  our  traffic  levels  to  those  in  other  studies,  we  have 
expressed  them  in  the  number  of  aircraft  handled  per  minute.  We  instructed  our  SMEs  to  create 
traffic  levels  that  reflected  situations  where  a  supervisor  would  be  about  to  change  the  staffing  in 
the  sector  to  only  an  R-side  ATCS  for  low  task  load  scenarios  and  to  a  three-person  team  (i.e., 
add  a  tracker)  for  high  task  load  scenarios.  Our  analysis  shows  that  our  task  load  manipulation 
worked  and  resulted  in  a  33%  increase  in  the  number  of  aircraft  handled  within  a  fixed  time 
interval. 
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To  compensate  for  the  variable  scenario  lengths,  we  calculated  the  number  of  altitude,  heading, 
and  speed  changes  per  aircraft.  This  approach  also  tends  to  compensate  for  differences  in  traffic 
volume.  Therefore,  changes  in  the  number  of  these  aircraft  maneuvers  reflect  a  change  in 
control  strategy  that  the  mere  change  in  number  of  aircraft  handled  cannot  explain.  Without 
expressing  aircraft  maneuvers  in  number  per  aircraft,  an  increase  in  traffic  task  load  will,  of 
course,  lead  to  an  increase  in  the  number  of  aircraft  maneuvers  within  a  scenario.  This  increase 
by  itself  may  increase  ATCS  workload  and  alter  other  behavioral  aspects.  Our  results  indicate, 
however,  that  ATCSs  use  more  speed  changes  per  aircraft  under  high  traffic  task  load  conditions. 
In  contrast,  ATCSs  used  fewer  altitude  changes  per  aircraft  when  traffic  task  load  increased. 

With  larger  traffic  volumes,  aircraft  have  less  open  airspace  in  which  to  maneuver,  and,  to 
compensate  for  this,  ATCSs  used  speed  instead  of  heading  changes.  ATCSs  often  indicate  that 
an  airspace  with  more  climbing  and  descending  aircraft  is  more  difficult  to  control  than  airspace 
with  mostly  en  route  aircraft  at  fixed  altitudes.  When  aircraft  make  a  vertical  transition,  ATCSs 
have  more  trouble  predicting  their  behavior  and  risk.  Under  high  task  load,  ATCSs  tried  to 
reduce  the  level  of  uncertainty  by  reducing  the  number  of  vertical  maneuvers  per  aircraft. 

The  complexity  and  handoff  efficiency  items,  when  examined  by  interval,  showed  significant 
effects  for  task  load  and  interval.  When  task  load  was  high,  more  altitude,  heading,  and  speed 
changes  occurred,  and  ATCSs  accepted  and  initiated  more  handoffs.  With  higher  task  loads, 
ATCSs  controlled  more  aircraft,  thus  more  control  actions  were  required,  as  indicated  by  the 
higher  numbers  for  the  complexity  and  handoff  efficiency  items.  As  time  increased,  the  number 
of  altitude  changes  gradually  decreased;  significantly  more  altitude  changes  occurred  during  the 
first  3-minute  interval.  At  the  beginning  of  the  scenario,  the  ATCS  had  to  work  traffic  that  was 
not  previously  controlled.  This  most  likely  contributed  to  the  higher  number  of  altitude  changes 
at  the  beginning  of  the  scenario  because  the  ATCS  needed  to  clear  the  planes  on  either  the  arrival 
or  departures  routes  to  the  proper  altitudes.  As  the  scenarios  progressed,  simulation  pilots 
generally  performed  more  heading  and  speed  changes,  whereas,  ATCSs  accepted  and  initiated 
more  handoffs.  ATCSs  in  our  experiment  sat  down  at  the  position  and  had  very  little  time  to 
take  over.  Early  in  the  scenarios,  ATCSs  maneuvered  aircraft  to  set  the  scenario  up  to  their 
liking.  While  they  were  catching  up,  they  built  a  bigger  plan,  likely  foreseeing  some  of  the 
future  situations.  This,  in  turn,  led  to  less  need  for  maneuvers  that  increase  uncertainty  such  as 
altitude  changes  later  in  the  scenario. 

3.6.2  ATCSs’  Interactions  with  the  DSR 


To  determine  how  ATCSs  interacted  with  the  DSR  interface,  we  used  the  SAR  tapes  collected 
during  the  experimental  simulation  runs.  With  the  NAS  Data  Analysis  and  Reduction  Tool 
(DART),  we  extracted  ATCS  input  messages  from  the  SAR  tapes,  fdtered  messages  from  the 
sector  under  study,  identified  whether  messages  came  from  the  R-  or  the  D-side,  and  stored  them 
in  text  files.  We  then  used  an  application  in  Labview  (National  Instruments  Corporation,  2000) 
to  parse  and  tally  the  messages  and  write  them  to  data  files  for  further  analysis. 

3 .6.2.1  Data  Analysis  and  Results 

Due  to  problems  with  SAR  tape  expiration  dates,  we  lost  tapes  for  the  last  four  participants  and 
had  missing  tapes  and  data  for  the  other  12  participants.  Because  of  the  substantial  loss  of  the 
data,  we  did  not  use  mean  replacement  but  structured  the  dataset  for  between-subject  analyses 
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using  an  unequal  N.  We  realize  this  analysis  does  not  take  advantage  of  eliminating  between 
subject  variance  and  incorporate  this  during  the  interpretation  of  the  statistical  analyses. 

We  divided  the  ATCS  interactions  into  several  groups  for  analysis.  We  performed  a  2  x  2  x  3 
(position  x  task  load  x  automation)  between-subjects  ANOVA  for  the  total  number  of  entries  into 
the  DSR  system.  We  grouped  the  number  of  flight  plan  readouts,  j-rings,  and  route  displays  as 
information  (Information  Pickup)  that  the  ATCSs  gathered  from  the  system.  The  number  of 
altitude  changes,  interim  altitudes,  removal  of  interim  altitudes,  and  route  changes  composed 
ATC  control  actions.  Station-keeping  activities  coded  from  the  DART  included  initiating  and 
accepting  handoffs  and  datablock  movement.  For  these  three  categories,  we  conducted  2x2x3 
(position  x  task  load  x  automation)  between-subjects  MANOVAs.  We  followed  any  significant 
multivariate  results  with  univariate  analyses  and  adjusted  the  alpha  level.  We  also  assessed  any 
results  with  a /7-value  of  .1  to  determine  if  anything  stood  out  that  may  be  of  interest  to  examine 
in  future  studies.  We  did  this  because  of  the  limitations  due  to  missing  data  and  using  between- 
subjects  analyses.  Appendix  BB  contains  the  means,  SDs,  and  ANOVA  tables. 

3.6.2.1.1  Total  Number  of  Entries 


Significant  main  effects  for  position  and  task  load  occurred  for  the  total  number  of  entries  made 
into  the  system  [F(l,52)  =  277.17, p  <  .001  and  F(l,52)  =  12.43,/?  <  .001,  respectively. 
Appendix  BB,  Table  BB-4,  Figure  84],  When  in  the  R-side  position,  ATCSs  total  number  of 
entries  was  higher  than  when  in  the  D-side  position.  ATCSs  made  more  entries  under  high  task 
load  conditions. 


Figure  84.  Total  number  of  ATCS  entries  by  position  and  task  load. 

3.6.2. 1.2  Information  Pickup 

The  MANOVA  for  the  information  pickup  items  showed  a  significant  effect  for  position  [A  = 
.48,  F(3,50)  =  17.70,/?  <  .001,  Table  BB-5].  The  adjusted  alpha  was  .017.  The  main  effect  of 
position  for  the  number  of  flight  plan  readouts  and  j-rings  was  significant  [F(l,52)  =  37.32,/?  < 
.001  and  F(l,52)  =  12.34,/?  <  .001,  Table  BB-6  and  Table  BB-7,  respectively].  R-side  ATCSs 
used  more  flight  plan  readouts  and  j-rings  (Figure  85  and  Figure  86,  respectively).  A  secondary 
trend  for  position  for  number  of  route  displays  showed  that  R-side  ATCSs  used  more  route 
displays  than  D-side  ATCSs  (Figure  87,  Table  BB-8). 
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D-side 


Figure  85.  Number  of  flight  plan  readouts  by  position. 


R-side  D-side 


Figure  86.  Number  of  J-rings  by  position. 


R-side  D-side 


Figure  87.  Number  of  route  displays  by  position. 

3.6.2.1.3  ATC  Control  Actions 

The  MANOVA  for  ATC  control  actions  showed  significant  effects  for  position  and  task  load  [A 
=  .23,  F( 4,49)  =  41.06,  p  <  .0001  and  A  =  .60,  F(4,49)  =  8.28 ,p  <  .001,  respectively,  Table 
BB-9].  Because  we  had  significant  multivariate  findings,  we  conducted  subsequent  univariate 
analyses  and  used  an  adjusted  alpha  of  .013.  We  found  a  secondary  trend  for  task  load  for  the 
number  of  hard  altitudes  entered  into  the  system  (Figure  88,  Table  BB-10).  ATCSs  entered  more 
hard  altitudes  into  the  system  under  high  task  load  conditions. 
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Figure  88.  Number  of  hard  altitudes  entered  by  task  load. 

The  ANOVA  for  entering  interim  altitudes  in  the  system  showed  significant  effects  for  position 
and  task  load  [F(l,52)  =  106.81  ,p  <  .001  and  F(l,52)  =  14.35,/?  <  .001,  respectively,  Figure  89, 
Table  BB-12].  A  secondary  trend  in  the  data  showed  an  interaction  between  position  and  task 
load.  Under  high  task  load  conditions,  R-side  ATCSs  entered  more  interim  altitudes;  the  effect 
for  task  load  diminished  this  for  the  D-side  ATCS. 


Low  T ask  Load  High  T ask  Load 


Figure  89.  Number  of  interim  altitudes  by  position  and  task  load. 

The  effect  for  position  in  the  removal  of  interim  altitudes  was  significant  [F(l,52)  =  64.24,  p  < 
.001,  Table  BB-13],  A  trend  for  automation  showed  more  interim  altitudes  removed  under 
limited  automation  conditions  than  no  or  full  automation.  In  addition,  the  interactions  between 
position  and  automation  and  task  load  and  automation  and  the  3 -way  interaction  were  secondary 
trends.  The  limited  automation  condition  seems  to  drive  the  interaction.  Under  low  task  loads 
and  limited  automation,  R-side  ATCSs  removed  more  interim  altitudes,  but  D-side  ATCSs 
removed  fewer  interim  altitudes  under  low  task  loads  and  limited  automation  (Figure  90  and 
Figure  91). 
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Figure  90.  The  interim  altitudes  removed  by  R-side  position,  task  load,  and  automation. 
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Figure  91 .  The  number  of  interim  altitudes  removed  by  D-side  position,  task  load,  and 
automation. 


The  multivariate  analysis  for  station-keeping  items  showed  significant  effects  for  position  and 
task  load  [A  =  .19,  F(3,50)  =  70.95,/?  <  .001  and  A  =.77,  F(3,50)  =  4.887,/?  <  .01,  Table  BB-14] 
Follow-up  univariate  analyses  showed  a  significant  effect  for  position  in  the  number  of  handoffs 
initiated  [F(l,52)  =  77.17,/?  <  .0001,  Table  BB-15].  R-side  ATCSs  initiated  more  handoffs 
(Figure  92). 


R-side  D-side 


Figure  92.  Number  of  handoffs  initiated  by  position. 
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The  main  effects  for  position  and  task  load  were  significant  for  the  number  of  handoffs  accepted 
[F(l, 52)  =  183. 73,  p  <.001  and  F(l, 52)  =  14.65,/?  <  .001,  respectively,  Table  BB-16],  R-side 
ATCSs  accepted  more  handoffs  and  more  handoffs  occurred  during  the  high  task  load  scenarios 
(Figure  93).  The  position  by  task  load  interaction  was  a  secondary  trend.  The  R-side  ATCSs 
accepted  more  handoffs  under  high  task  load  conditions,  but  the  effect  of  task  load  was 
diminished  for  the  D-side  ATCS. 


Figure  93.  Number  of  handoffs  accepted  by  position  and  task  load. 

The  effect  of  position  was  significant  for  the  number  of  datablock  movements  [,F(1,52)  =  59.66, 
p  <  .001,  Table  BB-17].  The  R-side  ATCS  moved  more  datablocks  (Figure  94).  A  secondary 
trend  for  task  load  showed  that  ATCSs  moved  more  datablocks  under  high  task  load  conditions 
(Figure  95).  A  secondary  trend  for  the  position  x  automation  interaction  showed  that  D-side 
ATCSs  moved  more  datablocks  under  low  automation  conditions  than  limited  or  full  automation 
conditions,  but  the  number  of  datablocks  moved  by  R-side  ATCSs  remained  relatively  constant. 


Figure  94.  The  number  of  datablock  movements  by  position  and  task  load. 
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Figure  95.  The  number  of  datablock  movements  by  position  and  automation. 

3 .6.2.2  Discussion 

We  used  between-subjects  multivariate  and  univariate  analyses  on  the  ATCS  interactions  with 
the  DSR  because  of  problems  with  substantial  missing  data.  By  conducting  the  between-subjects 
analyses,  we  found  significant  effects  for  position  and  task  load  on  most  ATCS  interactions  with 
the  DSR.  However,  we  expected  to  find  effects  of  automation.  The  lack  of  statistically 
significant  findings  for  automation  reflects  the  substantial  between-subject  variance  present  in 
the  analyses  and  highlights  the  strength  of  using  a  repeated  measures  design.  The  control 
strategies  used  by  ATCSs  differ  greatly  on  an  individual  basis  and  even  from  ARTCC  to 
ARTCC.  Therefore,  the  variability  between  ATCS  interactions  with  the  DSR  results  in  a  larger 
amount  of  error  variance  that  makes  it  more  difficult  to  assess  effects  for  variables.  Within  a 
repeated  measures  design,  each  participant  is  his  or  her  own  control  and  lowers  this  error 
variance.  With  this  in  mind,  we  presented  some  findings  with  a  larger  p- value  to  highlight 
anything  that  may  be  of  interest  to  examine  in  future  studies. 

The  effect  for  position  was  significant  across  the  ATCS  interaction  categories,  information 
pickup,  ATC  control  actions,  and  station  keeping.  The  R-side  ATCSs  worked  directly  with  the 
DSR  display  and  traffic  and  were  more  likely  to  seek  additional  information  and  perform  ATC 
actions  and  station-keeping  activities.  Task  load  also  had  an  impact  across  the  set  of  items.  As 
task  load  increased,  more  actions  were  necessary  to  control  traffic.  Although  not  statistically 
significant,  we  found  an  interaction  between  position  and  automation  for  the  number  of 
datablocks  moved.  Under  limited  or  full  automation  conditions,  D-side  ATCSs  moved  fewer 
datablocks,  but  the  number  of  datablocks  R-side  ATCSs  moved  remained  relatively  constant. 
This  may  imply  that  the  D-side  ATCSs  used  the  automation  and  that  this  drew  their  attention 
away  from  assisting  the  R-side  ATCS  in  the  station-keeping  activity  of  moving  datablocks. 
Interesting,  the  number  of  datablocks  moved  by  the  R-side  did  not  increase  during  the 
automation  conditions.  Perhaps  the  D-side  ATCS  was  using  the  automation  to  maneuver  aircraft 
and  that  less  datablocks  needed  to  be  moved  or  that  the  R-side  did  not  have  enough  time  to  move 
the  datablocks  and  just  ignored  the  task. 

Also  of  interest  are  the  relatively  low  numbers  of  actions  D-side  ATCSs  performed.  The  D-side 
ATCS  is  supposed  to  assist  the  R-side  ATCS;  however,  our  results  show  that  the  D-side  offers 
minimal  assistance  for  the  ATCS  interaction  items.  The  R-side  ATCS  performed  approximately 
300  control  entries  to  roughly  80  entries  made  by  the  D-side  ATCS.  The  D-side  made 
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substantially  fewer  entries  for  such  items  as  flight  plan  readouts,  route  displays,  interim  altitudes, 
handoffs  initiated  and  accepted,  and  datablock  movements.  This  finding  may  be  reflective  of 
individual  style  differences  and  differences  from  ARTCC  to  ARTCC.  Some  R-side  ATCSs 
prefer  to  make  control  entries  and  have  the  D-side  support  as  an  extra  pair  of  eyes.  In  this  case, 
the  R-side  ATCS  requests  minimal  control  entry  assistance  from  the  D-side  ATCS.  The 
responsibilities  of  the  D-side  ATCS  also  differ  from  facility  to  facility.  At  some  ARTCCs,  the 
D-side  ATCSs  are  more  active  with  control  entries  and  initiate  and  accept  handoffs  and  move 
datablocks.  At  other  facilities,  R-side  ATCSs  are  responsible  for  control  entries.  The 
interactions  of  the  ATCSs  seem  to  indicate  that  the  D-side  only  takes  care  of,  at  most,  25%  of  the 
observable  activities.  If  the  D-side  function  is  to  reduce  the  R-side  workload  and  increase  the  R- 
side  efficiency,  that  support  function  is  inefficient  in  the  configuration  we  tested  in  this 
experiment. 

3.7  Questionnaires 

We  used  two  types  of  self-report  questionnaires  adapted  from  previous  experiments.  The 
questionnaires  included  an  Entry  Questionnaire  (Appendix  K)  and  a  PSQ  (Appendix  H)  (Abbott 
Nataupsky,  &  Steinmetz,  1987;  Guttman  et  al.,  1995;  Sollenberger  &  Stein,  1995;  Stein,  1992;  ’ 
Willems  et  al.,  1999).  The  Entry  Questionnaire  contained  questions  concerning  demographic  * 
information.  The  PSQ  contained  questions  about  various  aspects  of  controlling  traffic  during  a 
scenario.  ~  & 

We  administered  the  Entry  Questionnaire  and  PSQ  in  paper  and  pencil  format  and  transcribed 
the  responses  into  a  spreadsheet.  We  created  three  data  sets.  Because  of  problems  with  missing 
data,  we  divided  the  data  sets  into  one  with  16  participants  and  one  with  the  last  6  participants  & 
for  the  PSQ  responses.  The  other  data  set  contained  data  from  the  Entry  Questionnaire. 

3.7.1  Entry  Questionnaire 

The  Entry  Questionnaire  contained  questions  about  participant  background,  demographics,  and 
importance  of  provided  airspace  and  aircraft  information.  We  have  included  the  information 
about  background  and  demographics  in  section  2. 1  to  describe  the  characteristics  of  our 
participants.  For  most  of  our  experiments,  we  asked  ATCSs  about  importance  of  airspace  and 
aircraft  information  because  we  anticipate  that  with  the  change  of  the  ATCS  working  position, 
some  of  that  information  may  change  or  disappear.  By  recording  this  information,  the  designers 
of  future  ATC  systems  have  a  baseline  of  what  information  ATCSs  feel  is  important. 

3.7.1 . 1  Data  Analysis  and  Results 

The  analysis  of  the  Entry  Questionnaire  data  consisted  of  the  calculation  of  means  and  SDs 
(Appendix  CC).  Table  CC  -1  presents  the  means  and  SDs  for  general  questions  from  the  entry 
questionnaire.  The  range  of  the  scale  was  1  to  10. 

Table  CC-2  presents  ATCSs’  ratings  for  the  importance  of  various  aircraft  information  sorted 
from  most  important  to  least  important.  ATCSs  rated  the  current  aircraft  location,  current 
altitude,  and  arrival  airport  (within  sector)  as  the  most  important  aircraft  information. 

Table  CC-3  presents  the  ratings  for  the  importance  of  radar  display  information.  ATCSs  rated 
sector  boundaries,  restricted  area  boundaries,  and  filter  settings  as  the  most  important. 


92 


3.7. 1.2  Discussion 

ATCSs  indicated  that  position,  altitude,  and  destination  are  most  important.  Other  information 
presented  in  the  aircraft  representation  can  be  derived  from  this  information  (i.e.,  speed  is  a 
change  of  position;  heading  changes  are  related  to  a  change  of  position). 


We  adapted  the  PSQ  from  Willems  et  al.  (1999),  and  it  provided  information  about  several 
aspects  of  ATC  during  a  particular  simulation  scenario  (Appendix  H).  The  PSQ  contained 
general  questions  about  the  simulation,  the  perceived  ATCS SA,  and  the  NASA  TLX  items. 
Here  we  only  discuss  the  items  that  do  not  provide  information  for  other  measurement 


constructs. 


We  conducted  a  MANOVA,  structured  as  a  2  x  2  x  3  (position  x  task  load  x  automation) 

repeated  measures  design,  for  the  realism  items.  Because  the  MANOVA  showed  statistical 

significance,  subsequent  analyses  included  ANOVAs  on  the  individual  variables.  We  conducted 
separate  2x2x3  (position  x  task  load  x  automation)  within-subjects  ANOVAs  for  participant 
performance,  difficulty,  ATWIT  and  oculometer  interference,  and  simulation  pilot  performance_ 
We  conducted  a  2  x  2  (position  x  task  load)  within- subject  ANOVA  for  effectiveness  of  the  DS 
in  resolving  conflicts  when  using  trial  flight  plans,  and  a  2  x  2  x  2  (position  x  task  load  x 
automation)  within-subjects  ANOVA  for  competence  using  the  DST.  Appendix  DD  contains  the 
means,  SDs,  MANOVA,  and  ANOVA  tables. 


3.7.2.1.1  Realism 

Two  items  composed  the  realism  category:  how  representative  the  scenario  was  of  a  typical 
workday  and  how  realistic  the  simulation  was.  The  MANOVA  showed  a  significant  effect  for 
task  load  [A  =  61  F(2  14)  =  4.54,  p  <  .05,  Table  DD-2],  An  alpha  level  set  at  .025  determined 
significant  results’for  subsequent  ANOVAs  of  each  measure.  ATCSs  indicated  that  the  low  task 
load  scenarios  were  more  representative  of  a  typical  workday  than  the  high  task  load  scenarios 
rF(l,15)  =  6.80, p  <  .05,  Figure  96,  Table  DD-4],  Using  a  more  liberal  alpha  level  of  .05,  we 
found  several  trends  in  the  data.  As  automation  increased,  the  perceived  representativeness  of 
the  scenarios  decreased  (Figure  96).  In  addition,  ATCSs  rated  the  simulation  less  realistic  when 
working  in  the  full  automation  condition  (Figure  97). 


Low  Task  Load  High  T ask  Load 

No  Automation  — ■ —  Limited  Automation 

Full  Automation 


Figure  96.  Representativeness  by  task  load  and  automation. 
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Figure  97.  Realism  by  automation. 


Position  and  task  load  influenced  perceived  control  quality  [F(l,15)  =  5.04, p  <  .05  and  F(l,15) 
=  36.08,  p  <  .0001 ,  respectively,  Table  DD-5].  ATCSs  perceived  they  had  better  performance 
when  controlling  traffic  on  the  R-side  and  under  low  task  load  conditions  (Figure  98). 


Figure  98.  Performance  by  position  and  task  load. 


3.7.2. 1.3  Difficulty 


Automation  had  an  effect  on  ATCSs’  perceptions  of  scenario  difficulty  (Figure  99).  As 
automation  increased,  the  perceived  difficulty  of  the  scenario  increased  [F(2,30)  =  4.04,  p  < 
.05,Table  DD-6],  Tukey  HSD  post  hoc  tests  showed  ATCSs  rated  the  scenarios  more  difficult 


No  Automation  Limited  Full  Automation 

Automation 


Figure  99.  Difficulty  by  Automation. 
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when  full  automation  was  present  than  when  they  had  only  limited  automation.  The  difficulty 
rating  did  not  differ  statistically  between  the  full  automation  and  no  automation  conditions  or 
between  the  limited  and  no  automation  conditions.  We  found  an  additional  effect  for  task  load  in 
the  six-participant  data  set.  When  the  task  load  was  high,  ATCSs  rated  the  scenario  more 
difficult  than  when  they  controlled  low  levels  of  traffic  [F(l,5)  =  16.47,/?  <  .01,  Figure  100, 
Table  DD-7], 


Low  Task  Load  High  Task  Load 

Figure  100.  Difficulty  by  task  load  (N=6). 

3. 7.2. 1.4  ATWIT  and  Oculometer  Interference 


Overall,  the  perceived  interference  of  the  ATWIT  device  was  low  (Figure  101).  The  device 
bothered  the  ATCSs  even  less  when  task  load  was  low  [iE7’(  1,15)  =  13.59,  /?  < .  01,  Table  DD-8]. 
The  task  load  x  automation  interaction  was  significant  [F(2,30)  =  4.94,/?  <  .05,  Table  DD-8]. 
The  degree  of  automation  had  an  impact  on  the  perceived  level  of  interference  when  task  load 
was  high  [F(2, 30)  =  5.04,/?  <  .05,  Table  DD-9].  When  full  automation  was  present,  ATCSs 
described  the  ATWIT  device  as  more  interfering  than  when  automation  was  limited  or  not 
present  at  all.  The  level  of  automation  did  not  have  an  effect  in  the  low  task  load  condition.  The 
results  from  the  data  set  of  the  last  six  participants  differed  slightly.  Although  overall  ratings  of 
ATWIT  interference  were  relatively  low,  the  experimental  conditions  had  a  strong  influence  on 
ATCSs’  perceptions.  A  significant  3-way  interaction  was  found  [F(2,10)  =  5.60,  /?  <  .05,  Table 
DD-9].  Examination  of  the  simple  effects  of  task  load  and  automation  within  the  two  positions 
showed  a  significant  effect  for  task  load  and  the  task  load  x  automation  interaction  for  the  R-side 
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Figure  101.  ATWIT  interference  by  task  load  and  automation. 
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•  •  rcvi  c\  _  m  n  <  noi  and  F(2  lO')  =  13.20,  p  <  .001,  respectively,  Figure  102,  Table 
D°D91  but  not  for  the  D-side  ATCS.  When  task  load  was  high,  R-side  ATCSs  perceived  it  to  be 

b"heDe®f of  the  limi,ed  automati°n  and 

no  automation  conditions  [F(2,10)  =  9M,p<  .01,  Table  DD-9], 


9  -- 

CO 

8 

OD 

G 

*4— * 

7 

6  -- 

CO 

QC 

5  -. 
4  .. 

e 

C3 

3  - 

.9- 

2  - 

*o 

1  - 

ts 

C3 

CL, 

0  - 

Low  Task  Load 

-No  Automation 
Full  Automation 


High  Task  Load 
.Limited  Automation 


Figure  102.  ATW1T  interference  by  R-side  position,  task  load,  and  automation  (N  6). 

Because  only  the  ATCS  in  wi*  — ng^ffic. 

experimental  conditions  did  not  influence  these  ratings. 

3.7.2. 1 .5  Responsiveness  of  the  Simulation  Pilots 

Ratines  for  the  responsiveness  of  the  simulation  pilots  decreased  when  the  task  load  was  high 
F(U5) =14.38, p<  .01,  Figure  103,  Table  DD-11].  No  other  effects  were  significant. 


Low  Task  Load 


High  Task  Load 


Figure  103.  Responsiveness  of  the  simulation  pilots  by  task  load. 

avoir,  Effectiveness  of  the  DST  in  Resolving  Conflicts  When  Using  Trial  Flight  Plans 

^onditiorCw^perfotme^a^x^  (p'o^idon^as^hjad^within-subj^ctsANOVA^'o  a^sTany 
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differences  in  the  effectiveness  of  the  DST  in  resolving  conflicts  when  using  this  feature. 
ATCSs  viewed  the  DST  as  more  effective  in  resolving  conflicts  when  performing  D-side  duties 
than  R-side  duties  [F(l,15)  =  16.37,/?  <  .01,  Figure  104,  Table  DD-12]. 


Figure  104.  Effectiveness  of  DST  by  position. 

3 .7.2. 1 .7  Competence  Felt  When  Using  the  DST 

A  2  x  2  x  2  (position  x  task  load  x  automation)  within-subjects  ANOVA  showed  that  ATCSs  felt 
more  competent  using  the  DST  when  the  traffic  task  load  was  low  than  high  [F(l,5)  =  9.31,/  < 
.01,  Figure  105,  Table  DD-13], 
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Low  Task  Load  High  Task  Load 

Figure  105.  Competence  felt  using  the  DST  by  task  load. 

3. 7.2.2  Discussion 

ATCSs  found  full  automation  conditions  most  difficult.  The  ATCS  participants  did  not  have 
experience  using  the  DST  prior  to  participating  in  the  study.  They  received  several  days  of  DST 
training  and  one  practice  day  with  the  DST  in  the  actual  experimental  environment.  They  did 
not  have  enough  time  to  automate  the  necessary  behaviors  required  to  operate  the  tool,  making  it 
more  difficult  for  them  to  use.  In  contrast,  they  were  much  more  familiar  working  in  an 
environment  with  flight  strips,  similar  to  the  no-automation  condition.  The  differences  in 
familiarity  between  use  of  the  flight  strips  and  use  of  the  DST  would  contribute  to  the  differences 
in  perceived  difficulty.  Further,  the  N=6  participant  data  set  showed  an  effect  for  task  load. 
ATCSs  viewed  the  high  task  load  conditions  as  more  difficult  than  the  low  task  load  conditions. 
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With  foil  automation  present,  ATCSs  indicated  that  simulations  were  less  realistic.  The 
participants  currently  work  in  a  setting  similar  to  the  no-automation  condition.  They  use  flight 
strips  and  do  not  have  DSTs;  therefore,  it  may  not  be  surprising  to  find  the  condition  with  foil 
automation  and  use  of  the  DST  rated  as  less  realistic  by  them. 

Even  though  ATCSs  indicate  that  traffic  levels  in  the  field  often  surpass  predicted  levels  for 
2005,  the  low  task  load  scenarios  in  this  experiment  were  closer  to  normal  traffic  than  high  task 
load  scenarios.  The  ATCSs  rated  the  low  task  load  scenarios  as  more  representative  and  realistic 
of  a  typical  workday.  This  finding  seems  congruent  with  what  the  ATCS  actually  experiences 
currently  in  the  field.  The  ATCS  participants  came  from  level  10,  11,  and  12  facilities  even 
though  we  had  requested  only  1 1  and  12s.  In  the  field  environment,  the  volume  of  traffic  a 
facility  gets  influences  the  rating  level  for  that  facility.  As  traffic  volume  increases,  the  rating 
level  of  the  facility  increases.  Even  at  the  highest-level  facilities,  a  traffic  “push”  or  peak  in 
traffic  volume  may  typically  last  60  minutes  and  usually  occurs  several  times  a  day  (A.  Nagy, 
personal  communication,  January  9,  2001).  Even  with  a  push  in  traffic,  the  amount  of  traffic  ’ 
fluctuates.  We  designed  the  scenarios  with  traffic  levels  that  an  ATCS  from  a  level  1 1  or  12 
facility  would  see;  however,  the  traffic  push  did  not  fluctuate  as  much  as  it  would  in  the  field. 
The  ATCS  participants  worked  the  scenario  traffic  for  at  least  30  minutes  at  a  constant  push.  We 
designed  the  low  traffic  scenarios  for  1 1  and  12  ATCSs,  and  even  the  low  traffic  was  more  than 
a  level  10  ATCS  would  see  in  the  field.  ATCSs  from  level  12  facilities  were  capable  of  handling 
the  scenario  traffic  levels  with  ease  and  indicated  that  they  had  expected  more  challenges.  In 
contrast,  ATCSs  from  level  10  facilities  found  the  scenarios  quite  difficult  and  found  ft  hard  to 
keep  up.  Future  traffic  levels  in  the  field  will  increase,  and  we  built  our  traffic  scenarios  to 
reflect  this.  Currently,  some  facilities  or  certain  sectors  within  facilities  already  experience  these 
high  volumes  of  traffic.  Because  the  scenarios  rotated  under  the  various  automation  conditions, 
the  effect  of  the  task  load  level  was  solely  responsible  for  the  differences  seen  in  the  ratings  of 
representative  and  realism  and  not  from  the  effect  of  a  particular  scenario. 

When  asked  about  their  performance,  ATCSs  indicated  that  they  controlled  traffic  better  when 
sitting  on  the  R-side  and  when  working  low  traffic.  R-side  ATCSs  have  direct  and  active  control 
of  the  traffic,  whereas  the  D-side  ATCSs  assist  the  R-side  and  do  not  directly  deal  with  the 
planes.  A  D-side  ATCS  is  in  a  more  monitoring  function,  therefore  feels  less  in  control.  This 
results  in  a  perception  of  a  lower  control  quality,  although,  as  a  team,  the  ATCSs  may  have 
controlled  excellently.  It  is  as  if  the  D-side  ATCSs  feel  that  they  can  help  more  if  they  could 
only  have  more  of  an  R-side  function.  In  debriefings  from  the  current  and  past  studies.  Certified 
Professional  Controllers  (CPCs)  stated  that  they  like  to  be  in  control  of  the  traffic  situation  and 
work  the  R-side  position  relative  to  the  D-side  position.  Our  participant  controllers  are  radar 
certified,  therefore,  they  are  used  to  doing  R-side  activities,  and  their  mindset  is  focused  on  more 
radar  tasks.  When  they  move  to  the  D-side  position,  they  are  still  aware  of  what  they  could  do  as 
an  R-side  and  have  a  sense  of  not  doing  as  much  as  their  capabilities.  The  caveat  is  that  all  of 
our  ATCSs  were  radar  certified.  In  addition,  the  low  traffic  task  load  scenarios  allowed  the 
ATCS  time  to  gain  the  “picture”  and  maintain  it. 

ATCSs  indicated  that  ATWIT  hardly  interfered.  ATCSs  felt  that  there  was  more  interference 
when  task  load  was  high  and  even  made  more  so  with  full  automation.  When  examining  the  6- 
participant  data  set,  the  3-way  interaction  was  significant  and  indicated  that  the 
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task  load  x  automation  interaction  was  significant  for  R-side  ATCSs  but  not  for  D-side  ATCSs. 
When  sitting  on  the  R-side,  the  highest  interference  from  the  ATWIT  device  occurred  when  task 
load  was  high  and  full  automation  was  present.  R-side  ATCSs  may  have  perceived  more 
interference  from  the  ATWIT  under  these  conditions  because  the  D-side  ATCS  directed  their 
attention  to  the  use  of  the  DST,  which  led  to  less  focus  on  the  usual  D-side  tasks  of  assisting  the 
R-side.  In  the  debriefings,  the  participants  confirmed  this  and  stated  that  they  rarely  looked  at 
the  radar  display  or  assisted  the  R-side  through  the  usual  D-side  responsibilities.  Because  the  R- 
side  ATCS  did  not  have  the  usual  assistance  from  the  D-side  ATCS,  the  R-side  ATCS  may  have 
been  more  sensitive  to  the  ATWIT  device  when  task  load  was  high.  In  addition,  the  ATWIT 
device  and  SAVANT  occurred  concurrently  and  resulted  in  a  blind  window  of  15  seconds  in 
which  the  ATCSs  could  not  view  the  radar  screen.  Under  high  task  load  traffic,  the  effect  of  the 
“blind”  window  would  be  stronger;  time  away  from  the  scope  would  be  increasingly  harder  with 
increasing  amounts  of  traffic. 

In  contrast  to  the  ATWIT,  ATCSs  did  not  rate  the  oculometer  as  more  interfering  in  one 
condition  versus  another.  The  ATCS  wore  the  oculometer,  which  provided  constant  discomfort 
but  did  not  require  specific  physical  responses  as  the  ATWIT  did. 

Even  though  both  ATCSs  used  the  DST,  once  at  the  R-side  they  put  less  confidence  in  it  than  on 
the  D-side.  The  R-side  ATCS  may  be  unaware  of  the  work  the  D-side  was  doing.  ATCSs 
indicated  in  the  debriefings  that,  when  on  the  R-side,  they  did  not  pay  attention  to  what  the  D- 
side  was  doing  with  the  DST.  The  D-side  ATCSs  used  the  DST,  and  the  R-side  ATCS 
controlled  traffic.  The  actual  use  of  the  DST  most  likely  led  to  perceptions  of  it  being  more 
effective.  The  ATCSs  felt  more  competent  using  the  tool  when  the  task  load  was  low.  We 
required  that  the  ATCS  participants  needed  to  be  novice  DST  users,  therefore  the  low  task  load 
scenarios  gave  the  participants  more  time  and  opportunity  to  work  with  the  DST  enabling  them 
to  feel  like  they  had  the  chance  to  fully  utilize  it. 

3.8  Subject  Matter  Expert  Rating  Forms 

Sollenberger,  Stein,  and  Gromelski  (1997)  developed  and  evaluated  a  method  to  assess  ATCS 
performance.  They  designed  a  rating  form  to  measure  the  effectiveness  of  new  or  enhanced 
ATC  systems  in  simulation  research.  The  rating  form  uses  an  8-point  format  and  a  comment 
section  for  each  of  the  questions.  Sollenberger  et  al.  showed  that  most  of  the  rating  scales  were 
very  reliable.  The  OTS  ratings  consisted  of  six  categories:  Maintaining  Safe  and  Efficient 
Traffic  Flow,  Maintaining  Attention  and  SA,  Prioritizing,  Providing  Control  Information, 
Technical  Knowledge,  and  Communication  related  questions  (Appendix  I). 

3.8.1  Data  Analysis  and  Results 

The  SME  completed  one  rating  form  for  each  team  of  ATCSs  per  condition.  We  only  had  one 
SME  available  for  ratings,  and  he  focused  mainly  on  the  R-side.  We  assumed  that  any  changes 
in  the  team  should  be  reflected  in  the  R-side.  We  averaged  the  two  observations  per  condition 
and  conducted  analyses  on  the  eight  teams.  We  analyzed  each  set  of  questions  separately  in  2  x 
3  (task  load  x  automation)  within-subjects  MANOVAs.  We  include  the  means,  SDs,  MANOVA, 
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and  ANOVA  tables  in  Appendix  EE.  Here  we  only  discuss  the  items  that  do  not  provide 
information  for  other  measurement  constructs. 


This  category  contained  three  items:  maintaining  separation  and  resolving  potential  conflicts, 
sequencing  arrival  and  departure  efficiently,  and  using  control  instructions  effectively  and 
efficiently.  The  MANOVA  results  showed  a  significant  effect  for  task  load  across  the  set  of 
DVs  [A  =  .16,  F(3,5)  =  8.72,  p  <  .05,  Table  EE-2,  Table  EE-3,  Table  EE-4,  and  Table  EE-5],  We 
did  not  find  any  significant  findings  for  automation.  Using  an  adjusted  alpha  of  .017,  subsequent 
ANOVAs  showed  that  ATCSs  sequenced  arrival  and  departure  aircraft  more  efficiently  in  the 
low  task  load  condition  and  maintained  separation  and  resolved  potential  conflicts  better  under 
low  task  load  conditions,  although  the  latter  finding  did  not  reach  statistical  significance  (i.e.,  it 
was  a  secondary  trend;  Figure  106). 


Figure  106.  Maintaining  safe  and  efficient  traffic  flow  by  task  load. 
3.8.1 .1.1  Prioritizing 


For  the  prioritizing  items,  taking  actions  in  an  appropriate  order  of  importance,  preplanning 
control  actions,  and  handling  control  tasks  for  several  aircraft,  the  MANOVA  showed  a 
significant  effect  of  task  load  [^  =  .17,  F(3,5)  =  7.96,  p  <  .05,  Table  EE- 10],  The  SME  rated 
preplanning  control  actions  higher  in  the  low  task  load  condition  [F(l,7)  =  31.29 ,p  <  .001, 
Figure  107,  Table  EE-12],  Using  a  more  liberal  alpha  of  .05,  secondary  trends  indicated  that 
ATCSs  took  actions  in  an  appropriate  order  of  importance  and  handled  control  tasks  for  several 
aircraft  better  under  low  task  load.  When  no  automation  was  present,  the  SME  rated  ATCSs 
lower  for  taking  actions  in  an  appropriate  order  of  importance  (Figure  108). 

Because  ATCSs  used  flight  strips  in  only  the  no-automation  condition,  we  used  a  one-way 
ANOVA  to  examine  the  effects  of  task  load.  Results  indicated  that  task  load  did  not  affect  the 
marking  of  flight  strips. 
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Figure  107.  Prioritizing  by  Task  Load. 
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Figure  108.  Taking  action  in  an  appropriate  order  of  importance  by  automation. 

3 .8 .1.1. 2  Providing  Control  Information 

The  SME  indicated  that  all  items  related  to  ATCSs’  providing  control  information  were  lower 
under  high  task  load  conditions  [A  =  .12,  F(3,5)  =  1 1.89,/?  <  .01,  Table  EE-15].  ATCSs 
provided  essential  ATC  information,  additional  ATC  information,  and  coordination  better  when 
controlling  traffic  in  the  low  task  load  scenarios  [F(l,7)  =  49.00 ,/?  <  .001,  F(l,7)  =  17.31,/?  < 
.01,  and  F(l,7)  =  10.86,/?  <  .05,  Table  EE-16,  Table  EE-17,  and  Table  EE-18,  respectively, 
Figure  109].  A  trend  indicated  that  ATCSs  received  higher  ratings  for  providing  additional 
information  when  they  were  in  the  limited-  or  full-  automation  conditions  relative  to  the  no¬ 
automation  condition  (Figure  110). 

3.8. 1.1.3  Technical  Knowledge 

Showing  knowledge  of  LOAs  and  SOPs,  showing  knowledge  of  aircraft  capabilities  and 
limitations,  and  showing  effective  use  of  equipment  composed  the  technical  knowledge  items. 
The  MANOVA  showed  a  significant  effect  of  task  load  [A  =  .14,  F(3,5)  =  9.76,  p  <  .05,  Table 
EE- 19].  ATCSs  demonstrated  more  effective  use  of  the  equipment  when  task  load  was  low 
[F(l,7)  =  30.77,/?  <  .001,  Figure  1 11,  Table  EE-22].  No  other  effects  were  significant. 
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Figure  109.  Providing  control  information  by  task  load. 


Figure  110.  Providing  additional  ATC  information  by  automation. 


Showing  Knowledge  of  Showing  Knowledge  of  Showing  Effective  Use 
LOAs  and  SOPs  Aircraft  Capabilities  of  Equipment 

and  Limitations 


□  Low  Task  Load  g  High  Task  Load 


Figure  111.  Technical  knowledge  by  task  load. 


3. 8. 1.1. 4  Communicatinf 


We  found  no  statistically  significant  effect  for  the  experimental  conditions  on  the 
Communications  items  at  the  multivariate  level.  A  primary  trend  showed  that  when  no 
automation  was  present,  an  increase  in  task  load  led  to  a  decrease  in  ratings  for  communicating 
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clearly  and  efficiently.  These  effects  diminished  in  the  two  automation  conditions  (Figure  112 
Figure  113,  respectively).  Secondary  trends  indicated  that  the  SME  rated  ATCSs  higher  for 
communicating  clearly  and  efficiently  and  listening  for  pilot  read  backs  and  requests  when  they 
were  in  the  full-  or  limited-  automation  conditions  relative  to  the  no-automation  condition 
(Figure  1 14).  Another  secondary  trend  for  using  proper  phraseology  showed  that  an  increase  in 
task  load  led  to  a  lower  rating  that  was  diminished  in  the  two  automation  conditions  (Figure  115 
and  Figure  1 16,  respectively). 


No  Automation  Limited  Automation  Full  Automation 
Figure  1 12.  Communicating  clearly  and  efficiently  by  low  task  load  and  automation. 
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Figure  113.  Communicating  clearly  and  efficiently  by  high  task  load  and  automation. 
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Figure  114.  Communicating  by  automation. 


No  Automation  Limited  Automation  Full  Automation 


Figure  115.  Using  proper  phraseology  by  low  task  load  and  automation. 
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Figure  1 16.  Using  proper  phraseology  by  high  task  load  and  automation. 
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3.8.1 .2  Discussion 


These  results  imply  that  with  the  assistance  of  automation,  the  D-side  was  able  to  free  some 
cognitive  resources  of  the  R-side.  This,  in  turn,  allowed  the  ATCS  to  provide  additional  control 
information  and  communicate  better.  Other  research  groups  have  used  cognitive  engineering 
techniques  to  design  future  ATC  working  positions  (Leroux,  1998).  Although  the  DST  used  in 
this  study  does  not  explicitly  assist  in  managing  the  ATCS’s  cognitive  resources,  it  seemed  to 
affect  the  distribution  of  the  use  of  these  resources  within  the  ATCS  team.  The  trends  further 
indicated  that  automation  led  to  the  ATCSs  gaining  a  better  focus  on  the  big  picture  and 
putting  control  items  in  the  appropriate  order  of  importance. 

Although  modest,  the  SME’s  ratings  indicated  better  ATCSs  performance  along  several  control 
actions  with  full  automation  present.  Please  note  that  the  automation  was  used  by  the  D-side  ^ 
ATCS  and  may  have  impacted  his  or  her  performance;  however,  due  to  limited  resources,  we  did 
not  have  the  ATC  SME  focus  primarily  on  the  D-side  for  OTS  ratings.  The  ratings  presented  in 
this  section  are  primarily  for  the  R-side  ATCS. 

The  data  clearly  show  that  increasing  task  load  decreased  ATCSs’  performance  across  the  OTS 
rating  form  items.  It  would  seem  that  an  increase  in  task  load  may  lead  the  ATCS  to  loose  focus 
on  the  big  picture.  This  leads  the  ATCS  to  implement  less  appropriate  planning  strategies  and  to 
communicate  less  effectively  as  he  or  she  tries  to  gain  the  picture. 

4.  GENERAL  DISCUSSION 

For  each  of  the  constructs  discussed  in  this  report,  we  present  a  description,  results,  and  a 
discussion  of  the  implications  of  these  results.  In  this  general  discussion,  we  place  these  results 
in  a  broader  framework,  presenting  the  effects  of  changes  in  automation,  task  load,  and 
responsibilities  (D-  vs.  R-side)  on  ATCSs.  However,  before  we  discuss  the  results  in  a  general 
context,  there  are  several  spin-offs  of  this  experiment  that  are  worth  mentioning  here: 

•  Creation  of  a  crosscutting  team  that  involves  test  and  evaluation,  design  and  engineering, 
simulation,  and  human  factors  groups  that  speak  a  common  language  and  work  together 
to  make  things  happen. 

•  Development  of  Generic  Center  Airspace  on  the  Host  Computer  System  (HCS)  and  the 
DST. 

•  Transfer  of  RDHFL  human  factors  instruments  to  operational  hardware  and  software. 

•  Development  of  a  new  S  A  assessment  instrument. 

The  existing  generic  center  airspace  used  for  testing  of  new  equipment  and  software  is  not  suited 
for  human-in-the-loop  experiments  that  look  at  concepts  from  a  human  factors  perspective.  The 
airspace  layout  and  naming  convention  makes  it  difficult  if  not  impossible  for  ATCSs  to  learn. 
The  ZCY  airspace  documentation  that  we  were  able  to  retrieve  dated  back  to  the  early  1970s. 

We  used  the  foundation  of  the  ZCY  adaptation,  wiped  the  airspace  clean,  and  built  the  Generic 
Center  Airspace  on  top  of  it. 
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The  creation  of  the  airspace  not  only  made  it  available  to  others  on  the  HCS  and  the  DST  but 
now  sets  the  standard  for  future  studies  (i.e.,  the  Generic  Center  Airspace  will  form  the 
foundation  for  future  simulations).  Although  we  can  create  new  sectors,  navigational  aids,  and 
so  on,  they  will  be  additions  to  the  Generic  Center  Airspace. 

Over  the  years,  we  have  developed  a  toolbox  of  human  factors  instruments  at  the  RDHFL  that 
we  have  used  extensively  on  our  in-house  simulator  platform.  This  experiment  resulted  in  a 
transition  of  most  if  not  all  of  these  instruments  to  an  environment  with  operational  hardware  and 
software.  The  successful  use  of  these  instruments  in  this  experiment  is  a  demonstration  that 
studies  that  involve  operational  hardware  and  software  can  use  our  instruments  as  well.  For 
operational  test  and  evaluation,  this  means  that  human  factors  measures  are  available  to  evaluate 
new  procedures  and  systems  that  we  have  tested  in  our  laboratory  settings. 

The  development  of  SAVANT  created  an  SA  assessment  tool  that  takes  the  advantages  of 
several  existing  instruments  and  ties  them  into  a  real-time  environment  while  taking  advantage 
of  having  the  actual  situational  data  available  to  probe  participants.  The  implementation  of 
SAVANT  was  a  technical  challenge  as  well;  our  programmers  needed  to  create  a  remote  display 
that  was  a  real-time  duplicate  of  the  DSR  display  at  the  sector. 

This  experiment  involved  the  operation  of  operational  hardware  and  software,  changes  in  NAS 
adaptations  that  require  in-depth  knowledge  of  the  NAS  architecture,  programming  across 
several  platforms,  and  transferring  data  using  multiple  protocols,  and,  on  top  of  all  this,  all 
systems  needed  to  work  in  a  controlled  experimental  fashion.  The  demonstration  that  this  can 
work  is  an  achievement  in  itself  and  shows  the  abilities  of  the  multidisciplinary  environment  of 
the  FAA  William  J.  Hughes  Technical  Center. 

4.1  Automation-Related  Observations 


We  have  seen  in  most  of  our  data  sets  that  with  increased  levels  of  automation,  D-side  ATCSs 
pulled  away  from  the  radar  display.  In  the  eye  movement  data,  we  have  objective  information  on 
how  much  time  the  D-side  ATCS  spent  looking  at  the  radar  display.  The  visual  scanning 
information  confirmed  what  other  data  sources  had  suggested,  that  is,  D-side  ATCSs  spent  less 
time  looking  at  the  radar  display  when  automation  levels  increased.  The  fact  that  the  D-side 
ATCS  divided  his  or  her  visual  attention  between  the  radar  display  and  the  DST  display  suggests 
that  they  also  spent  a  considerable  amount  of  time  in  transition  between  these  two  displays. 
During  this  transition  time,  the  D-side  ATCS  cannot  absorb  detailed  information. 

Recommendation:  To  provide  the  D-side  ATCS  with  information  about  the  current 
situation  and  to  reduce  the  amount  of  time  spent  in  transition  between  information 
displays,  we  should  investigate  if  we  can  integrate  data  from  automation  tools  into  a  radar 
display.  This  enhanced  radar  display  would  become  the  primary  information  display  for 
the  D-side  ATCS. 

In  this  study,  we  explained  to  our  participants  the  purpose  of  the  tool  and  emphasized  the 
strategic  nature  of  the  tool.  As  a  result,  we  observed  that  ATCSs  took  advantage  of  these 
capabilities  and  reached  out  to  adjacent  sectors.  It  is  interesting  to  note  that  when  we  discussed 
this  finding  with  ATCSs  from  ARTCCs  that  are  already  using  a  prototype  DST,  they  did  not 
agree  with  our  findings.  It  seems  that  in  the  field,  the  tool  is  much  less  used  in  the  strategic 
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fashion  for  which  we  had  trained  our  participants.  Instead,  ATCSs  wait  until  aircraft  are  under 
control  of  the  sector  before  taking  action.  As  mentioned  earlier,  this  may  be  the  result  of  the 
timing  of  the  assignment  of  a  D-side  ATCSs  to  a  sector. 

Recommendation:  The  balance  in  roles  and  responsibilities  between  ATC  sector  team 
members  has  evolved  over  many  years.  It  may  be  more  efficient  to  not  disturb  that  balance 
and  create  a  new  position  that  is  responsible  for  strategic  planning. 

When  we  provided  limited  automation,  ATCS  SA  was  lower  than  without  or  with  full 
automation.  The  D-side  ATCS  without  the  graphical  representation  of  the  DST  and  without  the 
familiar  flight  progress  strips  may  have  lost  some  SA  under  those  conditions.  The  R-side  ATCS 
in  turn  may  have  received  normal  D-side  assistance  during  simulations  without  automation  but 
less  effective  assistance  under  limited-automation  conditions.  This  in  turn  led  to  a  reduction  in 
SA  for  the  R-side  ATCS.  When  full  automation  was  available,  although  the  R-side  may  have 
lost  some  of  the  D-side  assistance  on  radar  tasks,  he  or  she  gained  assistance  of  the  D-side  on 
strategically  resolving  conflicts.  The  subjective  ratings  on  SA  for  potential  violations  showed 
that  ATCSs  felt  that  with  full  automation,  their  SA  was  lower  under  high  traffic  task  load 
conditions  for  which  we  also  found  an  indication  in  the  reduced  score  on  the  sector-based 
SAVANT  questions.  This  finding  shows  that  the  benefit  that  the  strategic  solution  provides 
under  low  traffic  task  load  conditions  no  longer  offsets  the  loss  of  D-side  assistance  on  radar 
tasks.  Within  a  sector  team,  a  delicate  balance  between  member  roles  and  responsibilities  exists. 

The  introduction  of  automation  may  seem  to  have  a  benefit  under  low  traffic  task  load  conditions 
but  may  disrupt  that  balance  when  ATCSs  work  under  high  traffic  task  load  conditions. 

Recommendation:  When  testing  the  benefits  of  automation  tools,  ATCSs  need  to  be  able  to 
work  in  team  configurations  that  reflect  their  normal  environment.  The  traffic  that  forms 
the  basis  for  these  tests  should  include  levels  that  one  expects  during  a  push  of  traffic. 

Recommendation:  The  lower  SA  found  for  ATCSs  using  limited  automation  may  indicate 
that  a  potential  loss  of  SA  may  occur  with  the  introduction  of  electronic  flight  strips.  It 
may  prove  to  be  beneficial  to  examine  the  impact  of  electronic  flight  strips  on  ATCS  SA 
within  a  controlled  lab  setting. 

We  found  that  under  full  automation  and  low  task  load,  ATCSs  had  longer  LL  communications 
than  under  high  task  load  conditions.  This  seems  to  reflect  that  the  D-side  ATCS  took  more  time 
to  communicate  with  adjacent  sectors  to  try  to  solve  potential  problems  strategically.  Under  high 
task  load  conditions,  the  D-side  ATCS  needed  to  revert  to  the  traditional  D-side  responsibilities 
that  do  not  include  the  use  of  the  DST.  Without  the  DST,  less  information  about  potential 
conflicts  is  available  and  communication  with  other  sectors  was  less. 

The  participants  in  this  study  had  no  previous  experience  with  the  DST.  During  the  first  week, 
we  trained  the  ATCS  on  the  concepts  and  use  of  the  DST.  Although  experts  in  their  field,  our 
participants  were  novice  users  of  the  DST.  The  ATCSs  indicated  that  simulations  with  full 
automation  present  were  less  realistic.  The  situation  in  the  field  most  resembles  the  simulation 
conditions  that  did  not  include  the  DST.  It  is  therefore  not  surprising  ATCSs  felt  that  the  full 
automation  condition  was  least  realistic  and  most  difficult.  The  perceived  difficulty  may  be 


107 


because  the  use  of  the  DST  was  new  and  therefore  involved  conscious  effort,  whereas  other 
overleamed  behaviors  are  automated  and  effortless.  Besides  increasing  the  perceived  difficulty 
ratings  for  the  D-side  ATCS,  this  also  may  have  led  to  an  indirect  increase  in  the  perceived 
difficulty  for  the  R-side  ATCS;  the  D-side  had  to  redirect  some  of  the  resources  normally  used  to 
support  the  R-side,  leaving  the  R-side  ATCS  with  less  assistance.  The  workload  ratings  and 
work  by  Bressolle  et  al.  (2000)  support  the  finding  that  the  introduction  of  automation  tools  may 
lead  to  an  increase  in  workload. 

Recommendation:  Additional  research  is  needed  into  how  long  an  increase  in  workload 
lasts  for  novice  DST  users  and  whether,  with  more  experience,  this  effect  diminishes.  More 
information  regarding  the  training  needs  of  the  ATCS  for  learning  DSTs  is  needed. 

In  an  ideal  environment,  the  advanced  automation  tools  will  adapt  to  the  amount  of  support  an 
operator  needs,  returning  some  of  the  responsibilities  to  the  operator  when  the  task  load  permits. 
AA  does  not  change  the  way  the  operator  works  but  assists  the  operator  with  his  or  her  tasks. 

The  implementation  of  the  DST  used  in  the  current  study  could  not  support  the  ATCS  in  an  AA 
environment  because  it  created  tasks  that  were  not  compatible  with  the  way  ATCSs  worked  in 
the  non-automated  environment.  In  an  environment  without  DSTs,  ATCSs  use  flight  progress 
strips  to  determine  the  potential  for  loss  of  separation.  Therefore,  AA  should  assist  ATCSs  in 
more  efficient  use  of  their  current  working  methods.  One  way  to  do  so  may  be  to  assist  ATCSs 
in  identifying  potential  conflicts.  In  the  DST  used  in  this  study,  the  electronic  equivalent  of  the 
flight  progress  strips  identified  the  number  of  potential  conflicts  by  aircraft  but  did  not  provide 
information  on  which  aircraft  these  potential  conflicts  were  with.  ATCSs  need  to  move  to  either 
a  textual  or  graphical  display  to  identify  conflict  pairs.  Under  high  task  load  conditions,  where 
the  automation  should  assist  ATCSs  without  interrupting  their  working  habits,  the  flight  progress 
equivalent  did  not  seem  to  provide  the  information  in  a  compatible  format.  In  the  current 
experiment,  we  saw  that  it  pulled  the  D-side  ATCS  away  from  his  or  her  role  as  a  radar 
associate,  leaving  the  R-side  ATCS  with  less  help  than  under  conditions  without  the  automation. 
The  DST  has  all  the  information  needed  to  assist  the  ATCSs  but  may  be  able  to  assist  better  by 
integrating  the  information  into  their  working  methods.  We  suggest  using  the  same  information 
provided  by  the  DST  and  presenting  it  in  a  format  that  is  more  compatible  with  ATCS  automated 
behaviors. 

Recommendation:  When  implementing  automation,  designers  should  give  special  care  to 
its  compatibility  with  ATCS  automated  behaviors  and  working  habits. 

One  of  the  observations  we  made  during  the  simulations  was  that  the  D-side  ATCS  spent  a  lot  of 
time  merely  separating  data  blocks  (i.e.,  preventing  data  block  information  to  overlap  and 
become  unreadable).  Even  though  this  may  not  be  a  very  difficult  task,  when  the  introduction  of 
automation  pulled  the  D-side  ATCS  away  from  the  normal  D-side  functions,  the  R-side  ATCS 
now  needed  to  separate  data  blocks  him-  or  herself,  leaving  less  time  and  fewer  resources  to 
focus  on  separating  aircraft.  ATCS  interaction  data  showed  that  R-side  ATCSs  did  not  move 
more  datablocks  under  full  automation;  however,  the  number  of  datablocks  moved  by  D-side 
ATCSs  decreased  under  the  automation  conditions.  Unfortunately,  ATCSs  currently  use  leader 
line  orientation  and  length  as  memory  joggers.  It  seems  that  introducing  automatic  data  block 
separation  and  an  alternative  to  indication  of  the  information  now  stored  in  the  orientation  and 
length  of  the  leader  line  will  greatly  reduce  the  workload  of  the  ATCSs. 
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Recommendation:  Provide  an  automatic  data  block  separation  tool  and  ways  to  store 
information  needed  to  remind  ATCSs  of  planned  or  executed  actions.  This  will  free  the  D- 
side  from  that  task  and  allows  the  D-side  to  concentrate  on  ATC-related  matters  without 
taking  away  assistance  an  R-side  ATCS  may  need. 

4.2  Task  Load-Related  Observations 


After  each  of  the  scenarios  that  included  automation,  we  asked  ATCSs  if  they  trusted  the 
information  that  the  tool  provided.  Under  high  task  load  conditions,  when  ATCSs  needed 
assistance  most,  they  indicated  that  they  trusted  that  information  the  least.  Wiener  (1988)  has 
found  that  operators  may  turn  automation  off  under  high  workload.  In  our  study,  ATCSs  may 
not  have  turned  off  the  automation  but  may  have  disregarded  the  information  and  reverted  to 
established  operational  procedures. 

Recommendation:  During  training,  we  must  make  ATCSs  aware  that  the  behavior  of  the 
tool  does  not  depend  on  the  traffic  task  load.  Training  must  make  the  ATCS  feel  so 
comfortable  with  the  tool  that  the  ATCS  has  realistic  expectations  of  where  the  tool  can 
assist  and  how  much  the  ATCS  can  trust  the  provided  information.  Training  must  also 
focus  on  reducing  ATCS  workload  while  they  become  proficient  using  the  DST.  When  a 
tool  like  a  DST  is  in  use,  a  change  in  formal  roles  and  responsibilities  may  be  necessary  to 
prevent  the  D-side  ATCS  to  revert  to  currently  established  procedures  that  prevent  the  use 
of  the  DST. 

The  ATCSs  themselves  and  the  OTS  raters  felt  that  an  increase  in  task  load  decreased  SA  and 
performance  in  general.  The  objective  SAGAT  measure  also  showed  decreases  in  SA,  although 
this  was  not  true  for  Level  2  SA.  With  more  aircraft  in  the  scenario,  ATCSs  may  have  needed 
more  cognitive  resources  than  available,  leading  to  a  loss  of  SA.  Interestingly,  we  found  that 
ATCSs  did  attempt  to  set  up  the  high  traffic  scenarios  to  become  more  structured,  potentially 
reducing  the  loss  of  SA.  With  an  increase  in  traffic,  ATCSs  used  more  speed  changes  and  less 
altitude  changes.  ATCSs  often  indicate  that  airspace  with  more  climbing  and  descending  aircraft 
is  more  difficult  to  control.  When  an  aircraft  makes  a  vertical  transition,  ATCSs  seem  to  have 
more  trouble  predicting  its  behavior  and  risk.  Under  high  traffic  levels,  ATCSs  tried  to  reduce 
the  level  of  uncertainty  of  the  situation.  In  the  current  system,  ATCSs  do  not  have  tools 
available  to  indicate  or  identify  aircraft  that  pose  a  high  risk.  Therefore,  ATCSs  need  to  monitor 
all  aircraft  in  the  sector.  It  is  likely  that  ATCSs  have  developed  strategies  that  distribute  their 
attention  in  such  a  way  that  high-risk  aircraft  receive  most  attention,  but,  under  high  traffic  task 
load  conditions,  they  seem  to  be  less  able  to  maintain  the  overall  picture.  Tools  that  distinguish 
between  aircraft  that  have  little  risk  of  separation  or  sequencing  problems  vs.  aircraft  that  are 
likely  to  get  into  situations  that  involve  increased  risk  situations  may  be  able  to  assist  ATCSs  to 
better  focus  their  attention  on  those  aircraft. 

Recommendation:  With  the  anticipated  increase  in  traffic  in  the  next  decade,  ATCSs  will 
have  increasing  difficulty  maintaining  good  SA.  Developing  tools  that  support  ATCSs  to 
focus  their  attention  on  aircraft  that  are  at  risk  to  get  into  potential  conflicts  or  otherwise 
complex  ATC  situations  may  allow  ATCSs  to  make  most  efficient  use  of  their  cognitive 
resources. 
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ATCSs  indicted  that  low  task  load  scenarios  were  closer  to  their  normal  traffic  levels.  In  the 
field,  a  traffic  push  may  typically  last  up  to  1  hour  and  occurs  several  times  per  day.  Even  within 
these  traffic  pushes,  the  amount  of  traffic  fluctuates.  In  this  experiment,  we  designed  the 
simulations  to  have  an  almost  constant  level  of  traffic  because  that  was  one  of  the  variables  we 
manipulated.  The  levels  of  traffic  in  our  experiment  may  not  reflect  current  conditions  in  the 
field.  However,  the  results  found  for  the  high  traffic  load  scenarios  may  give  us  insight  in  what 
to  expect  when  future  traffic  levels  further  increase.  Additional  tools  for  spacing  and  sequencing 
may  then  create  traffic  pushes  that  are  at  a  constant  level  and  last  for  an  hour  or  more.  As  one 
may  expect,  the  air-ground  communications  increased  in  number  and  decreased  in  duration  with 
an  increase  of  task  load.  Even  though  the  FAA  provides  ATCSs  with  tools  to  make  conflict 
detection  and  resolution  easier,  the  increase  in  traffic  will  still  require  ATCSs  to  verbally  confirm 
the  arrival  of  an  aircraft  in  the  sector  and  tell  the  aircraft  to  switch  their  radio  to  the  next  sector 
frequency.  Without  a  solution  to  reduce  the  necessity  to  verbally  acknowledge  the  entry  and  exit 
of  aircraft,  the  capacity  of  the  speech  channel  will  form  a  major  bottleneck  in  ATC. 

Recommendation:  Alternatives  to  verbal  acknowledgement  of  aircraft  entering  and 
leaving  the  sector  can  alleviate  the  task  load  on  the  speech  channel  and  free  ATCS 
cognitive  resources.  Implementation  of  data  functions  that  support  these  alternatives  will 
likely  increase  ATCS  capacity  to  handle  more  aircraft  in  the  sector. 

We  found  that  ATCSs  had  very  low  levels  of  Level  3  SA  (i.e.,  they  were  not  able  to  project 
future  situations).  Although  ATCSs  in  the  current  system  work  mostly  tactically  (i.e.,  within  a 
timeframe  of  3-5  minutes),  we  had  expected  that  the  use  of  the  DST  would  extend  this  horizon, 
resulting  in  differences  between  the  non-automated  and  automated  conditions.  One  would 
expect  that  with  more  experience  in  the  use  of  the  DST,  ATCSs  would  indeed  extend  their  time-  - 
horizon  and  become  more  strategic,  resulting  in  higher  Level  3  SA.  Although  the  data  may  be 
available  on  the  use  of  the  prototypes  that  are  in  operation  in  the  field,  there  are  no  studies 
available  that  show  this  shift  in  ATCS  SA. 

Recommendation:  Use  existing  operational  data  collected  since  the  implementation  of  the 
prototypes  in  the  field  to  determine  if  ATCSs  have  adopted  a  strategic  style  of  control.  If 
the  field  data  are  absent,  we  suggest  creating  an  experiment  that  uses  experienced  DST 
users  to  determine  if  they  have  better  Level  3  SA. 

4.3  ATCS  Position  Related  Observations 


In  the  field,  when  an  R-  and  a  D-side  work  a  sector,  the  D-side  is  responsible  for  coordination 
with  adjacent  sectors.  In  our  simulations,  coordination  using  the  LL  increased  with  an  increase 
in  task  load.  With  full  automation  available,  the  D-side  ATCS  had  the  opportunity  to  review  in 
more  detail  potential  conflicts  and  coordinate  control  actions  with  adjacent  sectors  before  aircraft 
entered  the  sector.  The  increase  in  duration  of  the  LL  communication  events  reflects  that  this 
happened  in  more  detail  when  task  load  was  low.  Another  explanation  for  the  difference  in  LL 
communication  durations  is  that  during  high  task  load  scenarios,  the  R-side  ATCS  required  full 
assistance  of  the  D-side.  For  this  experiment,  we  had  not  changed  the  roles  and  responsibilities 
of  the  D-side  ATCS.  Therefore,  the  primary  duty  of  a  D-side  ATCS  was  to  assist  the  R-side 
ATCS.  This  reflected  established  procedures  in  most  ARTCCs.  By  pulling  the  D-side  ATCS 
closer  into  the  support  of  the  tactical  management  of  traffic  in  the  sector,  the  D-side  had  less  time 
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to  coordinate  with  adjacent  sectors.  With  full  automation  present,  the  D-side  ATCS  still  tried  to 
resolve  potential  conflicts  through  coordination  with  adjacent  sectors  but  did  so  more  briefly. 

Without  providing  a  change  in  the  role  and  responsibilities  of  the  D-side  ATCS  when  providing 
automation  tools,  the  D-side  will  revert  to  established  procedures.  These  established  procedures 
will  pull  the  D-side  back  into  tactical  control  of  aircraft,  leaving  the  DST  not  used  to  its  full 
potential  when  it  would  be  able  to  assist  ATCSs  most.  This  is  not  to  say  that  the  R-side  ATCS 
does  not  need  the  D-side  assistance.  Under  the  high  task  load  conditions  that  we  tested  here,  the 
R-side  needed  someone  to  separate  data  blocks  and  an  extra  pair  of  eyes. 

In  the  field,  traffic  levels  fluctuate  and  are  relatively  low  before  a  push  in  traffic  occurs. 
Therefore,  one  can  bring  in  a  D-side  ATCS  before  an  anticipated  push  in  traffic.  The  D-side  can 
then  communicate  longer  with  adjacent  sectors  and  prepare  for  or  resolve  potential  conflicts. 

One  way  of  changing  the  role  of  the  D-side  may  be  to  implement  traffic  task  load  indices  that 
indicate  transitions  (e.g.,  relative  low  traffic  task  load)  with  a  transition  to  a  push  expected  in  30 
minutes.  During  those  30  minutes,  a  D-side  ATCS  could  focus  on  resolving  potential  conflicts 
and  be  more  distant  from  tactical  control.  This  could  reduce  the  peak  of  complexity  of  the  push. 
During  the  push  itself,  the  D-side  ATCS  pulls  back  from  a  strategic  into  a  more  tactical  mode 
and  fully  assists  the  R-side  ATCS. 

Recommendation:  When  a  DST  is  present  and  one  anticipates  a  push  in  traffic,  bring  a  D- 
side  ATCS  in  earlier,  so  that  that  position  can  make  maximum  use  of  the  strategic 
capabilities  of  the  DST. 

This  was  the  first  study  that  examined  eye  movements  in  D-side  ATCSs.  Our  results  indicate 
that  D-side  ATCSs  display  a  visual  scanning  behavior  that  is  very  different  from  R-side  ATCSs. 
As  one  would  expect,  the  D-side  ATCS  did  not  spend  as  much  time  looking  at  the  radar  display 
as  the  R-side.  We  also  found  that  the  D-side  ATCSs  focused  more  on  FDBs  than  R-side  ATCSs 
did  in  our  previous  studies.  Unfortunately,  we  did  not  collect  eye  movement  data  on  R-  and  D- 
side  ATCSs  simultaneously.  Therefore,  we  had  to  make  our  inferences  about  how  they  may  scan 
for  information  differently  from  results  obtained  from  several  studies. 

Recommendation:  When  providing  information  relevant  to  D-side  ATCSs,  it  may  be  most 
efficient  to  display  this  in  or  near  the  FDB. 

Recommendation:  To  better  understand  how  the  scan  for  information  differs  between  R- 
and  D-side  ATCSs,  we  should  measure  their  eye  movements  simultaneously. 

Although  the  D-side  ATCS  is  an  integral  part  of  the  sector  team,  our  participants  indicated  that 
they  felt  they  controlled  traffic  less  well  when  working  as  the  D-side  ATCS.  This  likely  reflects 
the  fact  that  the  D-side  does  not  directly  control  traffic.  Our  participants  were  radar-certified, 
ATCSs.  They  felt  they  could  have  assisted  better  when  at  the  radar  display  (i.e.,  more  actively 
involved  in  controlling  traffic).  When  working  the  D-side,  our  ATCS  participants  also  indicated 
that  high  traffic  task  loads  resulted  in  less  of  an  increase  in  workload  than  when  working  the  R- 
side.  Our  analyses  of  the  ATCS  interaction  with  the  DSR  interface  clearly  showed  that  R-side 
ATCSs  performed  the  majority  of  the  computer  entries.  This  combination  of  observations 
reflects  spare  capacity  within  the  sector  team  on  the  part  of  the  D-side.  When  assisting  the  R- 
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side,  the  D-side  may  be  able  to  take  on  more  but  is  not  able  to  do  so  because  of  the  division  of 
roles  and  responsibilities.  We  confirmed  this  feeling  of  being  less  able  to  directly  influence  the 
traffic  situation  by  objective  data  from  SAGAT.  SAGAT  results  indicated  that  for  a  given 
ATCS,  their  SA  was  lower  when  working  on  the  D-side  SA  compared  to  working  on  the  R-side. 

Recommendation:  More  research  in  the  roles  and  responsibilities  and  the  workload  of  D- 
side  ATCSs  is  necessary  to  determine  if  that  position  can  be  of  more  assistance  during  high 
traffic  situations  through  a  change  in  roles  and  responsibilities. 
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ACRONYMS 

AA 

Adaptive  Automation 

ANOVA 

Analysis  of  Variance 

ARTCC 

Air  Route  Traffic  Control  Center 

ATC 

Air  Traffic  Control 

ATCS 

Air  Traffic  Control  Specialist 

ATWIT 

Air  Traffic  Workload  Input  Technique 

C4T 

Controller-to-Controller  Communication/Coordination 
Taxonomy 

CA 

Conflict  Alert 

CAMI 

Civil  Aeromedical  Institute 

CPC 

Certified  Professional  Controller 

CRD 

Computer  Readout  Device 

D2 

Direct  To 

DART 

Data  Analysis  and  Reduction  Tool 

D-side 

Data-side 

DRA 

Data  Reduction  and  Analysis 

DRAT 

Data  Reduction  and  Analysis  Tool 

DSR 

Display  System  Replacement 

DST 

Decision  Support  Tool 

DV 

Dependent  Variable 

EDA 

En  Route  Descent  Advisor 

FAA 

Federal  Aviation  Administration 

FDB 

Full  Data  Block 

FL 

Flight  Level 

FPL 

Full  Performance  Level 

HCS 

Host  Computer  System 

HFE 

Human  Factors  Engineer 

I2F 

Integration  and  Interoperability  Facility 

IV 

Independent  Variable 

LL 

Landline 

LOA 

Letter  of  Agreement 

MANOVA 

Multivariate  Analysis  of  Variance 

MS  AW 

Minimum  Safe  Altitude  Warning 

NAS 

National  Airspace  System 

NASA 

National  Aeronautics  and  Space  Administration 

OE 

Operational  Error 

OTS 

Over-the-Shoulder 

PARR 

Problem  Analysis,  Resolution,  and  Ranking 

POG 

Point-of-Gaze 

PSQ 

Post-Scenario  Questionnaire 

PTT 

Push-to-Talk 

R 

Radar-side  Communication 

R-side 

Radar-side 

RADAR 

Radio  Detecting  and  Ranging 
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RDHFL 

Research  Development  and  Human  Factors  Laboratory 

RT 

Response  Time 

SA 

Situation  Awareness 

SAGAT 

SA  Global  Assessment  Technique 

SAR 

System  Analysis  Recording 

SART 

Situational  Awareness  Rating  Technique 

SAVANT 

SA  Verification  ANalysis  Tool 

SD 

Standard  Deviation 

SME 

Subject  Matter  Expert 

SOP 

Standard  Operating  Procedure 

SPAM 

Situation  Presence  Assessment  Method 

SWORD 

Subjective  Workload  Dominance 

TGF 

Target  Generation  Facility 

TLX 

Task  Load  Index 

TMA 

Traffic  Management  Advisor 

URET 

User  Request  Evaluation  Tool 

VOR 

Very  High  Frequency  Omnidirectional  Range 

vscs 

Voice  Switching  and  Communication  System 
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APPENDIX  A 


Participant  Recruitment  Form 


Air  Traffic  Control  Specialist 

Decision  Support  Automation  Research  Study 

William  J.  Hughes  Technical  Center 
Atlantic  City  International  Airport,  NJ 


In  this  study,  we  will  measure  air  traffic  control  performance  and  behavior  while  using  different 
levels  of  automation  under  two  levels  of  traffic.  This  is  fifth  in  a  series  of  studies  done  at  the 
Federal  Aviation  Administration  (FAA)  William  J.  Hughes  Technical  Center  (WJHTC)  Research 
Development  and  Human  Factors  Laboratory  (RDHFL)  to  measure  eye  movements. 

Improving  the  current  air  traffic  system  with  new,  automated  technologies  is  necessary  to  reduce 
or  manage  current  controller  workloads,  while  accommodating  continued  growth  in  air  traffic. 
We  will  investigate  how  task  load  and  automation  affect  controller  performance  and  behavior  in 
the  en  route  airspace. 

The  study  will  take  place  in  a  lab  area  of  the  Integration  and  Interoperability  Facility  (I2F).  We 
will  use  an  integrated  FAA  Target  Generation  Facility  (TGF),  Host,  and  Display  System 
Replacement  (DSR).  The  controller  environment  will  include  a  full  DSR  workstation  with  all 
operational  functions,  and  an  Air  Traffic  Workload  Input  Technique  (ATWIT)  device.  We  will 
make  use  of  a  generic  en  route  ARTCC  with  IFR  in  effect. 

The  Human  Factors  Laboratory  is  looking  for  16  volunteer  controllers  from  continental  US 
ARTCC  facilities.  The  time  requirement  for  this  experiment  is  two  weeks.  You  will  travel  on 
Monday  of  the  first  week  and  Friday  of  the  second  week.  From  Tuesday  through  Thursday  of 
the  first  week,  you  will  participate  in  training  in  a  classroom  setting  to  learn  how  to  use  the 
decision  support  tool.  On  Friday  of  the  first  week,  you  will  engage  in  a  series  of  simulations  to 
learn  the  simulated  airspace.  On  Monday  of  the  second  week,  you  will  train  on  simulations  on 
operational  hardware  and  software.  Finally,  from  Tuesday  through  Thursday  of  the  second  week 
you  will  participate  in  13  experimental  simulations. 

The  simulations  will  duplicate  operational  air  traffic  conditions.  You  will  interact  with 
simulation  pilots  and  control  simulated  air  traffic,  as  you  would  normally  do  in  the  field.  The 
simulations  will  have  two  levels  of  difficulty;  moderate  and  high  traffic  load.  During  the 
simulations,  you  will  either  control  traffic  as  you  would  currently  do  on  a  DSR  system,  or  use 
two  levels  of  automation.  The  first  level  will  require  only  limited  flight  strip  marking  and  will 
indicate  potential  conflicts  up  to  20  minutes  in  advance.  The  second  level  will  allow  the  D-side 
controller  to  create  trial  flight  plans  and  send  flight  plan  amendments  directly  to  the  Host. 
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During  half  of  the  simulations,  the  eye  movements  will  be  monitored.  A  small  camera  mounted 
on  a  headband  will  monitor  eye  movements.  An  invisible  beam  of  infrared  light  will  illuminate 
the  eye.  The  device  that  monitors  the  eye  movements  may  cause  some  discomfort.  The  skin 
area  under  the  headband  that  supports  the  device  may  show  some  redness  after  wearing  the 
device  for  the  duration  of  a  simulation.  The  intensity  of  the  infrared  beam  that  illuminates  the 
eye  is  about  one  thirtieth  of  the  intensity  expected  while  walking  outside  on  a  sunny  day  and 
should  not  cause  any  discomfort  or  risk  to  your  health. 

During  the  simulations  of  the  second  week,  we  will  record  video  images  and  audio.  These 
recordings  are  for  internal  use  only.  Two  researchers  will  be  present  to  code  the  communications 
between  the  R-  and  D-side  controllers.  During  these  simulations,  two  observers  will  conduct 
over-the-shoulder  ratings  -one  for  the  D-  and  one  for  the  R-side  controller. 

Your  only  direct  benefit  is  your  opportunity  to  participate  and  all  your  expenses  are  paid  under 
the  Federal  Travel  regulations.  The  benefit  for  air  traffic  controllers  derived  from  the  results  of 
this  experiment  may  include  a  better  understanding  of  why  operational  errors  occur  and  a  better 
understanding  of  how  controllers  reach  out  for  information. 

The  records  of  this  study  are  strictly  confidential,  and  you  will  not  be  identifiable  by  name  or 
description  in  any  reports  or  publications  about  this  study.  Photographs  and  audio  and  video 
recordings  will  be  made  during  the  study.  They  are  for  use  within  the  Research  and 
Development  Human  Factors  Laboratory  only.  Your  data  will  be  collected  by  code  number  and 
no  permanent  record  of  your  name  will  be  maintained. 

Schedule 

Two  controllers  will  visit  the  laboratory  each  week  beginning  on  or  about  February  7,  2000  until 
April  21, 2000.  Participants  will  travel  on  Monday  and  report  at  8:00  a.m.  on  Tuesday  for 
training.  The  training  sessions  will  conclude  on  Monday  afternoon  of  the  second  week.  Several 
30-minute  break  periods  will  be  taken  each  day  and  a  one-hour  lunch  period  will  be  taken  around 
midday.  The  simulation  will  stand  down  during  the  weeks  of  February  21st  for  Federal  Holidays. 

Research  psychologists  and  an  SATCS,  who  will  observe  the  participants  in  the  control  room, 
will  conduct  the  simulation.  A  voice  communication  link  to  another  room  will  allow  controllers 
to  issue  commands  to  trained  simulation  pilots.  Two  participants  will  operate  an  R-  and  D-side 
position.  During  the  simulations,  participants  will  alternate  between  R-  and  D-positions. 
Headsets  will  be  provided,  however,  participants  may  bring  their  own  headset  for  use  during  the 
experiment. 

Simulation  Procedures 

A  generic  ARTCC  sector  will  be  used.  The  sector  is  designed  for  easy  learning  in  a  short  period 
of  time.  The  sector  consists  of  easily  remembered  fix  names,  airports,  and  simplified  operating 
procedures. 

Participants  will  complete  a  background  questionnaire  at  the  beginning  of  the  experiment.  A 
subject  matter  expert  will  provide  an  airspace  and  equipment  briefing.  Participants  will  control 
practice  scenarios  to  ensure  familiarity  with  the  airspace  and  simulation  equipment.  Participants 


A-2 


will  then  control  experimental  scenarios.  After  each  experimental  scenario,  participants  will 
complete  a  Post-Scenario  Questionnaire.  Following  the  last  simulation  run  participants  will 
complete  a  final  questionnaire,  and  will  be  asked  for  detailed  comments  about  the  experiment. 

An  automated  data  collection  system  will  record  important  simulation  events  and  produce  a  set 
of  system  effectiveness  measures,  including  safety,  capacity,  efficiency,  and  controller  workload. 
The  simulation  will  be  video  recorded  for  research  purposes  only.  No  video  records  will  be 
released.  An  SATCS  will  make  over-the-shoulder  observations  during  each  simulation. 

Rights  of  Participants 

Participation  in  this  study  is  strictly  voluntary  and  the  privacy  of  participants  will  be 
protected.  No  individual  names  or  identities  will  be  recorded  or  released  in  any  reports. 
Strict  adherence  to  all  Federal,  Union,  and  ethical  guidelines  will  be  maintained 
throughout  the  study.  The  purpose  of  the  study  is  to  scientifically  assess  the  previously 
cited  concepts,  not  to  evaluate  the  individual  controllers. 

Minimum  Standards  for  Participants 

Participants  will  be  current  level  5  Radar  controllers  with  valid  medical  certificates.  Single  vision 
glasses  may  be  used  but  contact  lenses  are  not  acceptable.  Staff  and  supervisors  are  not 
acceptable. 

Travel 

Travel  and  overtime  expenses,  if  needed,  will  be  borne  by  the  Technical  Center.  Fund  codes  will 
be  provided  when  personnel  are  identified.  Sending  facilities  or  regions  will  write  travel  orders 
using  Technical  Center  fund  sites. 

Points  of  Contact 


Your  support  is  important  to  the  success  of  this  project,  and  your  cooperation  will  be  greatly 
appreciated.  If  you  have  any  additional  questions,  then  please  do  not  hesitate  to  contact  me. 

Earl  S.  Stein,  Ph.D.,  Engineering  Research  Psychologist,  Technical  Project  Lead,  (609)  485- 
6389,  ACT-530,  Bldg.  28,  Research  Development  and  Human  Factors  Laboratory,  FAA 
William  J.  Hughes  Technical  Center,  Atlantic  City  International  Airport,  NJ  08405. 

You  may  also  contact  Mr.  Tony  Buie,  SATS,  Project  Controller  subject  matter  expert,  (609)  485- 
4869. 
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APPENDIX  B 


Informed  Consent  Form 


I, _ ,  understand  that  the  Federal  Aviation  Administration 

sponsors  and  Ben  Willems  direct  this  study,  entitled  “Decision  Support  Automation  Research 
(DSAR)  in  the  En  Route  ATC  Environment  I.” 

Nature  and  Purpose 

I  will  volunteer  as  a  participant  in  the  project  above.  The  purpose  is  to  explore  active 
controllers’  use  of  different  levels  of  automation.  The  time  requirement  for  this  experiment  is 
four  days.  I  will  travel  on  Monday  and  Friday.  On  the  2  test  days  of  the  experiment,  I  will 
participate  in  6  practice  and  13  experiment  simulations  of  45  minutes  each. 

Experimental  Procedures 

During  half  of  the  simulations,  the  movements  of  my  eyes  will  be  monitored.  A  small  camera 
mounted  on  a  headband  will  monitor  my  eye  movements.  An  invisible  beam  of  infrared  light 
will  illuminate  my  eye. 

The  simulations  will  mimic  operational  air  traffic  conditions.  I  will  interact  with  simulation 
pilots  and  control  simulated  air  traffic  like  I  would  normally  do  in  the  field. 

Discomforts  and  Risks 


The  device  that  monitors  the  eye  movements  may  cause  some  discomfort.  The  skin  area  under 
the  headband  that  supports  the  device  may  show  some  redness  after  wearing  the  device  for  the 
duration  of  a  simulation.  The  intensity  of  the  infrared  beam  that  illuminates  the  eye  is  about  one 
thirtieth  of  the  intensity  expected  while  walking  outside  on  a  sunny  day  and  should  not  cause  any 
discomfort  or  risk  to  my  health. 

Benefits 

I  understand  that  the  only  direct  benefit  to  me  is  to  participate  in  research  in  Atlantic  City,  NJ. 

The  benefit  derived  from  the  results  of  this  experiment  for  controllers  may  include  a  better 
understanding  of  why  operational  errors  occur,  which  could  lead  to  new  ways  to  assist  ATC 
students. 

Participant’s  Responsibilities 

During  the  experiment,  it  will  be  my  responsibility  to  control  the  simulated  air  traffic  as  if  I  was 
controlling  traffic  at  my  home  facility.  I  will  answer  any  questions  asked  during  the  experiment 
to  the  best  of  my  abilities.  I  will  not  discuss  the  content  of  the  experiment  with  anyone  until  the 
completion  of  the  experiment. 
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Participant’s  Assurances 


I  understand  that  my  participation  in  this  study  is  voluntary.  Ben  Willems  has  adequately 
answered  any  questions  I  have  about  this  study,  my  participation,  and  the  procedures  involved.  I 
understand  that  Ben  Willems  will  be  available  to  answer  any  questions  concerning  procedures 
throughout  this  study.  I  understand  that  if  new  findings  develop  during  the  course  of  this 
research  that  may  relate  to  my  decision  to  continue  to  participation,  I  will  be  informed. 

I  have  not  given  up  any  of  my  legal  rights  or  released  any  individual  or  institution  from  liability 
for  negligence. 

I  understand  that  records  of  this  study  are  strictly  confidential,  and  that  I  will  not  be  identifiable 
by  name  or  description  in  any  reports  or  publications  about  this  study.  Photographs  and  audio 
and  video  recordings  are  for  use  within  the  Research  and  Development  Human  Factors 
Laboratory  only.  Any  of  the  materials  that  may  identify  me  as  a  participant  cannot  be  used  for 
purposes  other  than  internal  Research  and  Development  Human  Factors  Laboratory  without  my 
written  permission. 

I  understand  I  can  withdraw  from  the  study  at  any  time  without  penalty  or  loss  of  benefits  to 
which  I  may  be  entitled.  I  also  understand  that  the  researcher  of  this  study  may  terminate  my 
participation  if  he  feels  this  to  be  in  my  best  interest. 

If  I  have  questions  about  this  study  or  need  to  report  any  adverse  effects  from  the  research 
procedures,  I  will  contact  Ben  Willems  at  (609)  485-4191  during  Monday  through  Friday  or  at 
(609)  404-1650  in  the  evening  or  on  weekends. 

I  may  also  contact  Dr.  Earl  Stein  (609)  485-6389,  the  Air  Traffic  Human  Factors  Technical  Lead, 
at  any  time  with  questions  or  concerns. 

I  have  read  this  consent  document.  I  understand  its  contents,  and  I  freely  consent  to  participate 
in  this  study  under  the  conditions  described.  I  have  received  a  copy  of  this  consent  form. 

Research  Participant:  _ Date:  _ 

Investigator:  _ Date:  _ 

Witness: _ Date: 
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DST  Display  Windows 
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Figure  D-l.  Plans  display  window  from  DST. 
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Figure  D-2.  Departure  list  display  window  from  DST. 
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Figure  D-3.  Response  display  window  from  DST. 
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Implementation  of  Savant 

SAVANT’s  display  closely  matched  the  DSR  display.  We  presented  the  ATCS  with  several 
queries  and  recorded  the  controllers’  responses  by  having  them  click  on  an  aircraft.  Figure  E-l 
presents  the  differences  between  the  representations  between  the  DSR  and  the  SAVANT  display. 
SAVANT  recorded  objects  the  ATCS  chooses  as  the  correct  response.  All  ATCS  entries  carried 
a  time  stamp. 

SAVANT  ran  on  a  separate  computer,  one  for  the  D-side  and  one  for  the  R-side  controller.  The 
computer  display  presented  a  replication  of  the  airspace  including  targets  and  data  blocks. 
SAVANT  indicated  targets  of  interest  on  the  screen  by  highlighting  them  in  a  separate  color  and 
with  portions  of  their  data  blocks  hidden.  Participants  responded  to  queries  about  the  status  of 
the  targets.  They  responded  directly  on  the  computer  by  clicking  on  the  target  that  has  the 
highest  values  for  the  queried  variable.  When  SAVANT  started,  the  computer  screen  was  blank. 
Each  session  lasted  15  seconds  (3  seconds  to  display  the  question  and  9  seconds  to  respond). 

The  airspace  boundaries,  airways,  Very  High  Frequency  Omnidirectional  Range  (VORs),  etc. 
were  static  and  can  be  read  in  at  the  beginning  of  the  simulation.  The  aircraft  information  was 
overlaid  at  the  time  of  the  query  session.  The  aircraft  messages  in  the  NAS  are  the  1525 
messages.  We  then  needed  to  reconstruct  the  aircraft  representations  and  mask  some  information 
based  on  the  type  of  query  we  wanted  to  present. 

SAVANT  uses  the  ODS  Toolbox  (Orthogon,  1999)  as  the  human-computer-interface.  SAVANT 
obtained  part  of  the  data  for  the  replication  of  the  DSR  screen  from  the  target  generation  facility 
(TGF).  SAVANT  needed  other  data  to  correctly  place  data  blocks.  That  data  included  the 
orientation  and  length  of  the  leader  line  relative  to  the  raw  radar  return,  the  history  trail,  and  the 
vector  line.  A  token  ring  sniffer  listened  in  on  two  of  the  three  DSR  token  rings  and  presented 
the  DSR  messages  on  a  socket.  Middleware  grabbed  the  appropriate  messages  and  presented  it 
to  DESIREE.  Figure  E-l  presents  an  example  of  both  the  DSR  and  the  SAVANT  representation 
of  an  aircraft  pair.  The  implied  query  in  this  particular  case  would  be  “Which  aircraft  is  higher?” 
The  following  sections  will  briefly  discuss  the  implementation  of  the  sniffer,  middleware,  and 
DESIREE. 
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Figure  E-l.  Comparison  of  DSR  and  SAVANT  aircraft  representation  for  situation  awareness 

queries. 
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Figure  E-2.  Schematic  layout  of  DSAR1  hardware  and  software. 


Sniffer 


The  first  component  is  a  custom  Token  Ring  server  (TRSRV)  that  obtains  the  HGS/DSR  data 
and  forwards  it  to  a  middle-ware  component  that  processes  the  data,  and  then  passes  the 
information  to  a  back-end  display  processor.  This  data  is  identified  in  Interface  Control 
Document  NAS-IR-2 104-4001  (FAA,  1999). 

The  HCS  and  DSR  transfer  data  in  data  structures  (packets)  called  LIU  messages  or  System 
Requests  (SRs).  The  HCS  sends  the  data  through  the  HCS  channel  attachment  to  the  Local 
Communications  Network  (LCN)  Interface  Unit  (LIU).  This  RISC  6000  system  processes  the 
SRs  and  forwards  them  across  the  LCN,  the  token  ring  Broadcast  (B)  and  token  ring  Point-to- 
Point  (P)  rings,  to  the  appropriate  consoles  based  on  transmission  class  and  destination  name. 

The  data  that  originate  at  the  console  systems  travel  the  reverse  path  to  the  HCS. 

The  TRSRV  monitors  the  token  rings  in  the  promiscuous  mode  so  all  LAN  traffic  is  available  to 
the  server.  We  then  filter  the  SRs  according  to  a  pre-selected  subset  of  SRs  that  SAVANT  will 
use.  The  TRSRV  encapsulates  these  selected  SRs  in  UDP/IP  packets  and  forwards  them  to  an 
active  client,  the  middle-ware  component. 

The  TRSRV  starts  when  a  client  sends  an  arbitrary  packet  to  UDP  port  3900.  It  then  forwards 
the  selected  SR  packets  to  that  client.  The  client  can  terminate  the  TRSRV  operation  by  exiting. 
This  will  cause  the  TRSRV  to  receive  an  ICMP  “connection  refused”  status,  which  is  the  signal 
to  terminate  token  ring  monitoring.  If  the  TRSRV  receives  a  request  on  UDP  port  3900  from 
another  client  while  it  is  already  in  dialogue  with  a  current  client,  all  packets  will  be  forwarded  to 
the  new  client  exclusively  from  that  time. 

The  SAVANT  tool  currently  uses  a  subset  of  the  available  SRs:  1524,  1509,  1525,  and  0758. 

The  Add  Target  Position  (#1524)  SR  contains  primary  and  beacon  radar  for  display  on  the  Host 
Situation  Display  at  DSR  R-positions.  Also  the  Process  Limited  Data  Block  (#1509)  SR 
contains  Limited  Data  Blocks  for  display  on  the  Host  Situation  Display  at  DSR  R-positions.  In 
addition  the  Process  Track  Data  Group  Block  (#1525)  SR  contains  Full  Data  Blocks  for  display 
on  the  Host  Situation  Display  at  DSR  R-positions.  And  to  complete  the  subset  the  Process 
Operator  Host  Command  (#0758)  SR  contains  an  input  to  the  Host  from  a  DSR  R,  D,  or  A- 
position.  This  subset  of  SRs  can  convey  the  air  traffic  control  situation  as  presented  to  the 
controller,  and  the  controller  responses  as  the  controller  responds  to  the  situation. 

Middleware 


Purpose 

The  middleware  serves  two  purposes.  First,  it  provides  DESIREE  with  the  data  to  allow  a 
replication  of  the  aircraft  displayed  on  DSR.  Secondly,  it  records  the  data  needed  to  fully 
replicate  the  aircraft  displayed  on  DSR  for  posttest  data  reduction. 

Implementation 

The  middleware  process  uses  JAVA/CORBA/OrbixWeb,  running  on  a  Sun  workstation  under 
Solaris  2.7.  DESIREE  launches  the  middleware  process.  The  middleware  is  an  invoked  process 
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for  DESIREE.  The  middleware  then  connects  to  the  sniffer  by  sending  an  arbitrary  “wake-up” 
packet  to  UDP  port  3900. 

Procedure 


After  the  sniffer  wakes  up,  it  puts  the  filtered  SR  data  on  the  UDP  port.  The  middleware 
receives  the  filtered  SR  data  from  the  UDP  port.  The  data  format  on  the  UDP  port  is  similar  to 
the  format  specified  in  NAS-IR-21 04-400 1 .  However,  the  Full  Data  Block  (FDB)  message  1 525 
changes  to  message  1672  on  the  HCS  portion  of  the  DSR  token  ring.  The  difference  between  the 
1672  and  the  1525  message  is  that  the  1672  message  carries  FDB  data  for  one  sector  only,  while 
the  1525  message  carries  FDB  data  for  multiple  sectors. 

To  provide  DESIREE  with  data,  the  middleware  picks  up  data  packets  from  the  UDP  port, 
populates  CORBA  objects  with  the  data  received  from  the  sniffer,  and  DESIREE  picks  up  the 
objects. 

To  record  for  posttest  data  reduction,  the  middleware  writes  the  data  to  disk  in  XML  format. 

The  development  team  chose  the  XML  format,  because  XML  parsers  are  readily  available.  The 
file  names  are  date  and  time  stamped  to  the  actual  local  time.  The  recorded  data  allows  the  eye 
movement  data  reduction  software  to  correlate  point  of  gaze  data  with  the  position  of  aircraft 
radar  returns  and  data  blocks.  A  separate  software  package  uses  the  XML  files  to  transform  the 
data  into  a  format  that  is  compatible  with  the  existing  eye  movement  data  reduction  and  replay 
software  packages. 

DESIREE 


DESIREE  is  the  RDHFL  front  end  to  our  in-house  ATC  simulator.  Although  DESIREE  uses  the 
ODS  Toolbox  (Orthogon,  1999),  it  has  adopted  a  CORBA  environment.  In  the  implementation 
of  SAVANT,  DESIREE  incorporated  the  middleware  component,  a  file  reader  component,  and 
two  display  components. 

The  file  reader  component  enabled  researchers  to  create  text  files  that  contained  events  with  the 
corresponding  event-time.  These  events  included  the  swapping  of  displays,  presentation  of  a 
question,  highlighting  of  objects,  and  hiding  of  object  elements.  DESIREE  swapped  [need  a 
picture  here  to  show  the  physical  layout  of  the  DSR/Host/SAVANT/URET  setup]  the  DSR  and 
URET  displays  for  SAVANT  displays  by  controlling  a  wide  bandwidth  matrix  switch  through  a 
serial  port  connection. 

DESIREE  received  the  SR  data  by  populating  aircraft  objects  through  its  middleware 
component.  It  displayed  the  aircraft  objects  in  real  time  through  one  of  the  display  components 
on  the  same  Sony  high  resolution  (2000  x  2000  pixel)  display.  The  controllers  used  these  objects 
to  respond  to  our  queries.  The  other  display  component  recorded  the  current  displayed  objects 
to  a  file  on  disk. 
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ATWIT  Instructions 

ATWIT  instructions  given  before  calibration  of  the  oculometer. 

One  purpose  of  this  research  is  to  obtain  an  accurate  evaluation  of  controller  workload.  By 
workload,  we  mean  all  the  physical  and  mental  effort  that  you  must  exert  to  do  your  job.  This 
includes  maintaining  the  “picture,”  planning,  coordinating,  decision  making,  communicating, 
and  whatever  else  is  required  to  maintain  a  safe  and  expeditious  traffic  flow.  Every  five  minutes 
the  ATWIT  device,  located  to  the  side  of  the  radar  display,  will  emit  a  brief  tone  and  ten  buttons 
will  appear.  The  buttons  will  remain  visible  for  only  a  limited  amount  of  time.  Tell  us  how  hard 
you  are  working  by  pushing  the  buttons  numbered  from  1  to  10  on  the  ATWIT. 

I  will  review  what  these  buttons  mean  in  terms  of  your  workload.  At  the  low  end  of  the  scale  (1 
or  2),  your  workload  is  low  -  you  can  accomplish  everything  easily.  As  the  numbers  increase, 
your  workload  is  getting  higher.  Numbers  3,  4,  and  5  represent  increasing  levels  of  moderate 
workload  where  the  chance  of  error  is  still  low  but  steadily  increasing.  Numbers  6,  7,  and  8 
reflect  relatively  high  workload  where  there  is  some  chance  of  making  errors.  At  the  high  end  of 
the  scale  are  numbers  9  and  10,  which  represent  a  very  high  workload,  where  it  is  likely  that  you 
will  have  to  leave  some  tasks  unfinished. 

All  controllers,  no  matter  how  proficient  and  experienced,  will  be  exposed  at  one  time  or  another 
to  all  levels  of  workload.  It  does  not  detract  from  a  controller’s  professionalism  when  he 
indicates  that  he  is  working  very  hard  or  that  he  is  hardly  working.  Feel  free  to  use  the  entire 
scale  and  tell  us  honestly  how  hard  you  are  working.  Do  not  sacrifice  the  safe  and  expeditious 
flow  of  traffic  in  order  to  respond  to  the  ATWIT  device.  Remember,  your  workload  rating 
should  not  reflect  how  much  you  are  working  during  the  course  of  the  scenario.  Instead,  your 
rating  should  reflect  how  much  workload  you  are  experiencing  during  the  instant  when  you  are 
prompted  to  make  the  rating. 

Do  you  have  any  questions  about  using  the  ATWIT  device? 
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APPENDIX  G 


ATCS  Roles  and  Responsibilities 


Radar 

Radar  Associate  (RA) 

Flight  Data  (D) 

Non-Radar 

Ensure  separation 

Ensure  separation 

Operate  interphones 

Ensure  separation 

Initiate  control  instructions 

Initiate  control  instructions 

Assist  the  RA-position  in 
managing  flight  progress  strips 

Initiate  control  instructions 

Monitor  and  operate  radios 

Operate  interphones 

Receive/process  and  distribute 
flight  progress  strips 

Monitor  and  operate  radios 

Accept  and  initiate 
automated  handoffs 

Accept  and  initiate  automated 
handoffs,  and  ensure  radar  position  is 
made  aware  of  the  actions 

Ensure  flight  data  processing 
equipment  is  operational 

Accept  and  initiate  transfer 
of  control,  communications, 
and  flight  data 

Assist  the  RA  position  with 
non-automated  handoff 
actions  when  needed 

Assist  the  R-side  position  by 
accepting  or  initiating  automated 
handoffs  which  are  necessary  for  the 
continued  smooth  operation  of  the 
sector,  and  ensure  that  the  R-side  is 
made  immediately  aware  of  any 
action  taken 

Request/receive  and 
disseminate  weather, 

NOT  AM’s,  NAS  status,  traffic 
management,  and  Special  Use 
Airspace  status  messages 

Ensure  computer  entries  are 
completed  on  instructions 
or  clearances  issued  or 
received 

Assist  the  RA  position  in 
coordination  when  needed 

Coordinate  including  point  outs 

Manually  prepare  flight 
progress  strips  when 
automation  systems  are  not 
available 

Ensure  strip  marking  is 
completed  on  instructions 
or  clearances  issued  or 
received 

Scan  radar  display. 

Correlate  with  flight 
progress  strip  information 

Monitor  radios  when  not  performing 
higher  priority  duties 

Enter  flight  data  into  computer 

Facilities  utilizing  nonradar 
positions  may  modify  the 
standards  contained  in  the 
radar  associate 

Ensure  computer  entries  are 
completed  on  instructions  or 
clearances  you  issue  or 
receive 

Scan  Flight  Progress  Strips. 

Correlate  with  radar  data. 

Forward  flight  data  via 
computer 

Ensure  strip  marking  is 
completed  on  instructions  or 
clearances  you  issue  or 
receive 

Manage  Flight  Progress  Strips. 

Assist  facility/sector  in 
meeting  situation  objectives 

Adjust  equipment  at  R-side 
to  be  usable  by  all  members 
of  the  team 

Ensure  computer  entries  are 
completed  on  instructions  issued  or 
received  by  the  R-side  when  aware  of 
those  instructions. 

The  R-side  shall  not  be 
responsible  for  G/G 
communications  when 
precluded  by  VSCS  split 
functionality 

Ensure  strip  marking  is  completed  on 
instruction  issued  or  received  by  the 
R-side  when  aware  of  them. 

Adjust  equipment  at  RA-position  to 
be  usable  by  all  members  of  the  team 
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APPENDIX  H 


Post-Scenario  Questionnaire 


Please  complete  the  following: 


I.  NASA-TLX 


1. 

Circle  the  number  that  best  describes  the  mental 
demand  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

2. 

Circle  the  number  that  best  describes  the  physical 
demand  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

3. 

Circle  the  number  that  best  describes  the  temporal 
demand  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

4. 

Circle  the  number  that  best  describes  your 
performance  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

5. 

Circle  the  number  that  best  describes  your  effort 
during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

6 . 

Circle  the  number  that  best  describes  your  level  of 
frustration  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

II.  Scenario 


7.  Circle  the  number  below  that  best  describes  how 
hard  you  were  working  during  this  scenario. 

extremely 

hard 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Not 

hard 

8.  Circle  the  number  that  best  describes  how  well  you 
controlled  traffic  during  this  scenario. 

extremely 

poorly 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Extremely  1 

well 

9.  Circle  the  number  that  best  describes  your  overall 
situational  awareness  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Extremely 

high 

lO.Circle  the  number  that  best  describes  your 
situational  awareness  for  current  aircraft 
locations  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Extremely 

high 

1 1. Circle  the  number  that  best  describes  your 
situational  awareness  for  projected  aircraft 
locations  during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Extremely 

high 

12. Circle  the  number  that  best  describes  your 
situational  awareness  for  potential  violations 
during  this  scenario. 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Extremely 

high 

13.Circle  the  number  that  best  describes  how  difficult 
this  scenario  was. 

extremely 

difficult 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

easy 

14.Circle  the  number  that  best  describes  how 
competent  you  felt  using  the  decision  support 
tool(s). 

extremely 

incompetent 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

competent 

15. Circle  the  number  that  best  describes  how 

predictable  the  behavior  of  the  decision  support 
tool(s)  was. 

extremely 

unpredictable 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

predictable 

16. Circle  the  number  that  best  describes  how  well  the 
decision  support  tool(s)  predicted  the  outcome  of 
separation  strategies. 

Extremely 

poorly 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

well 

17.Circle  the  number  that  best  describes  how  effective 
the  decision  support  tool(s)  was  in  resolving 
conflicts  when  using  trial  flight  plans. 

extremely 

ineffective 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

effective 

18. Circle  the  number  that  best  describes  your  view  of 
the  capability  of  the  tool(s)  to  predict  future 
conflicts. 

Extremely 

unable 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

able 

19.Circle  the  number  that  best  describes  the  technical 
knowledge  incorporated  in  the  tool. 

extremely 

ignorant 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

knowledgeable 

20. Circle  the  number  that  best  describes  your  overall 
confidence  in  the  decision  support  tool(s). 

No 

confidence 

1 

2 

3 

4 

5 

6 

7 

8 

9  10 

complete 

confidence 

H-l 


III.  Simulation 


21  .Circle  the  number  that  best  describes  how  realistic  extremely  1  2  3  4  5  6  7  8  9  10 

the  simulation  was.  unrealistic 

22.  Circle  the  number  that  best  describes  how  Not  1  2  3  4  5  6  7  8  9~  10 

representative  the  scenario  was  of  a  typical  representative 

workday. 

23. CircIe  the  number  that  best  describes  if  the  no  interference  12345678910 

ATWIT  device  interfered  with  controlling  traffic. 

24. Circle  the  number  that  best  describes  if  the  no  interference  1  2  3  4  5  6  7  8  9  10 

oculometer  interfered  with  controlling  traffic. 

25.  Circle  the  number  that  best  describes  how  well  the  extremely  i  2  3  4  5  6  7  8  9  10 

simulation-pilots  responded  to  clearances  and  poorly 

callbacks. _ 

26.  Do  you  have  any  other  comments  about  your  experiences  during  the  simulation? 


extremely 

realistic 

extremely 

representative 

extreme 

interference 

extreme 

interference 

Extremely 

well 


APPENDIX  I 


OTS  Rating  Forms 


INSTRUCTIONS  FOR  QUESTIONS  1-20 

This  form  was  designed  to  be  used  by  instructor  certified  ATC  specialist  to  evaluate  the 
effectiveness  of  controllers  working  in  simulation  environments.  Observers  will  rate  the 
effectiveness  of  controllers  in  several  different  performance  areas  using  the  scale  show  below. 
When  making  your  ratings,  please  try  to  use  the  entire  scale  range  as  much  as  possible.  You  are 
encouraged  to  write  down  observations  and  you  may  make  preliminary  ratings  during  the  course 
of  the  scenario.  However,  we  recommend  that  you  wait  until  the  scenario  is  finished  before 
making  your  final  ratings.  The  observations  you  make  do  not  need  to  be  restricted  to  the 
performance  areas  covered  in  this  form  ands  may  include  other  areas  that  you  think  are 
important.  Also,  please  write  down  any  comments  that  may  improve  this  evaluation  form.  Your 
identity  will  remain  anonymous;  so  do  not  write  your  name  on  the  form. 


Rating 

Label  Description 

1 

Controller  demonstrated  extremely  poor  judgment  in  making  control  decisions  and  very  frequently  made  errors  . 

2 

Controller  demonstrated  poor  judgment  in  making  some  control  decisions  and  occasionally  made  errors 

3 

Controller  make  questionable  decisions  using  poor  control  techniques  which  led  to  restricting  the  normal  traffic  flow 

4 

Controller  demonstrated  the  ability  to  keep  aircraft  separated  but  used  spacing  and  separation  criteria  which  was  excessive 

5 

Controller  demonstrated  adequate  judgment  in  making  control  decisions 

6 

Controller  demonstrated  good  judgment  in  making  control  decisions  using  efficient  control  techniques 

7 

Controller  frequently  demonstrated  excellent  judgment  in  making  control  decisions  using  extremely  good  control  techniques 

8 

Controller  always  demonstrated  excellent  judgment  in  making  even  the  most  difficult  control  decisions  while  using 
outstanding  control  techniques 

I.  MAINTAINING  SAFE  AND  EFFICIENT  TRAFFIC  FLOW 


1.  Maintaining  Separation  and  Resolving  Potential  Conflicts 

-  using  control  instructions  that  maintain  appropriate  aircraft  separation 

-  detecting  and  resolving  impending  conflicts  early 

-  recognizing  the  need  for  speed  restrictions  and  turbulence  separation 

1 

2  3 

4  5 

6 

7 

8 

2.  Sequencing  arrival  and  Departure  Aircraft  Efficiently 

-  using  efficient  and  orderly  spacing  techniques  for  arrival  and  departure 
aircraft 

-  maintaining  safe  arrival  and  departure  intervals  that  minimize  delays 

1 

2  3 

4  5 

6 

7 

8 

3.  Using  Control  Instructions  Effectively/Efficiently 

-  providing  accurate  navigational  assistance  to  pilots 

-  avoiding  clearances  that  result  in  the  need  for  additional  instructions  to  handle 
aircraft  completely 

-  avoiding  excessive  vectoring  or  over-controlling 

1 

2  3 

4  5 

6 

7 

8 
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II.  MAINTAINING  ATTENTION  AND  SA 


4.  Maintaining  Awareness  of  Aircraft  Positions  1 

-  avoiding  fixation  on  one  area  of  the  radar  scope  when  other  areas  need 
attention 

2 

3 

4 

5 

6 

7 

8 

-  using  scanning  patterns  that  monitor  all  aircraft  on  the  radar  scope 

5.  Ensuring  Positive  Control  1 

-  tailoring  control  actions  to  situation 

2 

3 

4 

5 

6 

7 

8 

-  using  effective  procedures  for  handling  heavy,  emergency,  and  unusual  traffic 
situations 

6.  Detecting  Pilot  Deviations  from  Control  Instructions  1 

-  ensuring  that  pilots  follow  assigned  clearances  correctly 
*  correcting  pilot  deviations  in  a  timely  manner 

2 

3 

4 

5 

6 

7 

8 

7.  Correcting  Own  Errors  in  a  Timely  Manner  1 

-  acting  quickly  to  correct  errors 

-  changing  an  issued  clearance  when  necessary  to  expedite  traffic  flow 

2 

3 

4 

5 

6 

7 

8 

III.  PRIORITIZING 

8.  I  aking  Actions  in  an  Appropriate  Order  of  Importance  I 

-  resolving  situations  that  need  immediate  attention  before  handling  low  priority 
tasks 

2 

3 

4 

5 

6 

7 

8 

-  issuing  control  instructions  in  a  prioritized,  structured,  and  timely  manner 

9.  Preplanning  Control  Actions  I 

-  scanning  adjacent  sectors  to  plan  for  future  and  conflicting  traffic 

-  studying  pending  flight  strips  in  bay 

2 

3 

4 

5 

6 

7 

8 

10.  Handling  Control  Tasks  for  Several  Aircraft  1 

-  shifting  control  tasks  between  several  aircraft  when  necessary 

-  avoiding  delays  in  communications  while  thinking  or  planning  control  actions 

2 

3 

4 

5 

6 

7 

8 

1 1 .  Marking  Flight  Strips  while  Performing  Other  Tasks  1 

-  marking  flight  strips  accurately  while  talking  or  performing  other  tasks 

-  keeping  flight  strips  current 

2 

3 

4 

5 

6 

7 

8 

IV.  PROVIDING  CONTROL  INFORMATION 

12.  Providing  Essential  ATC  Information  1 

-  providing  mandatory  services  and  advisories  to  pilots  in  a  timely  manner 

-  exchanging  essential  information 

2 

3 

4 

5 

6 

7 

8 

13.  Providing  Additional  ATC  Information  1 

-  providing  additional  services  when  workload  is  not  a  factor 

-  exchanging  additional  information 

2 

3 

4 

5 

6 

7 

8 

14.  Providing  Coordination  1 

-  providing  effective  and  timely  coordination 

-  using  proper  point-out  procedures 

2 

3 

4 

5 

6 

7 

8 
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V.  TECHNICAL  KNOWLEDGE 


15.  Showing  Knowledge  of  LOAs  and  SOPs 

-  controlling  traffic  as  depicted  in  current  LOAs  and  SOPs 

-  performing  hand-off  procedures  correctly _ 

16.  Showing  Knowledge  of  Aircraft  Capabilities  and  Limitations 

-  avoiding  clearances  that  are  beyond  aircraft  performance  parameters 

-  using  appropriate  speed,  vectoring,  and/or  altitude  assignments  to  separate 

aircraft  with  varied  flight  capabilities _ 

17.  Showing  Effective  Use  of  Equipment 

-  updating  data  blocks 

-  using  equipment  capabilities  _ 

““  '  — ~  VI.  COMMUNICATING 

1 8.  Using  Proper  Phraseology 

-  using  words  and  phrases  specified  in  ATP  71 10.65 

-  using  ATP  phraseology  that  is  appropriate  for  the  situation 

-  avoiding  the  use  of  excessive  verbiage _ 

19.  Communicating  Clearly  and  Efficiently 

-  speaking  confidently  and  at  the  proper  volume  and  rate  for  pilots  to 
understand 

-  speaking  fluently  while  scanning  or  performing  other  tasks 

-  clearance  delivery  is  complete,  correct  and  timely  _ 

20.  Listening  for  Pilot  Readbacks  and  Requests 

-  correcting  pilot  readback  errors 

-  processing  requests  correctly  in  a  timely  manner 

-  acknowledging  pilot  or  other  controller  requests  promptly _ 


APPENDIX  J 


SA  Global  Assessment  Technique  Questions 


1 .  Enter  the  location  of  all  aircraft  by  indicating: 

-  the  aircraft  in  track  control 

other  aircraft  in  the  sector 

aircraft  that  will  be  in  track  control  in  the  next  2  minutes 

2.  Enter  the  aircraft  callsign  (for  the  aircraft  indicated  on  the  sector  map  in  question  1). 

3.  Enter  the  aircraft  altitude  (for  the  aircraft  indicated  on  the  sector  map  in  question  1). 

4.  Enter  the  aircraft  ground  speed  (for  the  aircraft  indicated  on  the  sector  map  in  question  1). 

5.  Enter  the  aircraft  heading  (for  the  aircraft  indicated  on  the  sector  map  in  question  1). 

6.  Enter  the  aircraft’s  next  sector  (for  the  aircraft  indicated  on  the  sector  map  in  question  1). 
(insert  list  of  sector  names  here  as  choices  followed  by  “landing  in  sector”) 

7.  Enter  the  aircraft’s  current  direction  of  change  in  each  column  (for  the  aircraft  indicated  on 
the  sector  map  in  question  1). 

Altitude  Change  Turn 

climbing  right  turn 

descending  left  turn 

level  straight 

8.  Enter  the  aircraft  type  (for  the  aircraft  indicated  on  the  sector  map  in  question  1). 

9.  Enter  the  aircraft’s  activity  in  the  sector  (for  the  aircraft  indicated  on  the  sector  map  in 
question  1). 

En  route 

inbound  to  airport 
outbound  from  airport 

10.  Which  pairs  of  aircraft  have  lost  or  will  lose  separation  if  they  stay  on  their  current  assigned 
courses? 

1 1 .  Which  aircraft  have  been  issued  clearances  that  have  not  been  completed? 

12.  Did  the  aircraft  receive  its  clearance  correctly  (for  each  entered  in  question  1 1)? 

13.  Which  aircraft  are  currently  conforming  to  their  clearances  (for  each  entered  in  question  1 1)? 

14.  Which  aircraft  will  be  handed  off  to  another  sector/facility  in  the  next  2  minutes? 

15.  Enter  the  aircraft  that  are  not  in  communication  with  you. 

16.  Enter  the  aircraft  that  will  violate  special  airspace  separation  standards  if  they  stay  on  their 
current  assigned  paths. 

17.  For  which  aircraft  is  weather  currently  impacting  or  will  be  impacting  on  in  the  next  5 
minutes? 
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APPENDIX  K 


Entry  Questionnaire 


Note:  We  provided  space  for  comments  after  each  question  and  at  the  end  of  the  questionnaire. 

Please  complete  the  following: 


1 .  What  is  your  age  in  years?  - years 

2.  Are  you  wearing  corrective  lenses  during  this  experiment?  □  Yes  □  No 

3.  How  many  years  have  you  actively  controlled  traffic?  _ years 

4.  How  many  years  have  you  controlled  traffic  at  your  current  facility?  _ years 

5.  How  many  months  in  the  past  year  have  you  actively  controlled  traffic?  . . . . months 

6.  What  is  your  current  position  as  a  controller?  □  Developmental  □  Full  □  Other: 


Performance 

Level 

7.  Please  list  other  facilities  where  you  have  worked:  _ _ 


8.  Do  you  search  the  DSR  display  in  one  special  way  for  information?  □  Yes  □  No 
If  it  depends  on  certain  factors,  what  are  they? 


9.  Please  circle  the  number  that  best  describes 
your  current  skill  as  a  controller. 

not 

skilled 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

skilled 

10. Please  circle  the  number  that  best  describes 
your  motivation  to  participate  in  this  study. 

not 

motivated 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

motivated 

1  l.How  often  do  you  use  vertical  separation? 

never 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

always 

12. How  often  do  you  use  vectoring  for 

never 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

always 

separation?  I 

13. How  often  do  you  use  speed  control  for 

never 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

always 

separation? 

14. Please  circle  the  number  that  best  describes 
your  experience  with  video  games. 

Not 

experienced 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

experienced 

Please  circle  the  number  that  best  describes  the  importance  of  the  following 

aircraft  information. 

15.  Aircraft  CallSign 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

16.  Aircraft  Type 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

17.  Aircraft  Beacon  Code 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

18.  Controller  Ownership 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

19.  Entry  Altitude 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

20.  Entry  Airspeed 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

21.  Entry  Fix 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

22.  Exit  Altitude 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 
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23. Exit  Airspeed 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

24.  Exit  Fix 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

25.  Arrival  Airport  (within  sector) 

extremely 

low 

I 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

26.  Departure  Airport  (within  sector) 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

27.  Current  Altitude 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

28. Current  Airspeed 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

29.  Current  Heading 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

30.  Current  Aircraft  Location 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

31.  Most  Recently  Assigned  Altitude 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

32.  Most  Recently  Assigned  Airspeed 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

33. Most  Recently  Assigned  Heading 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

34.  Aircraft  Holding/Spinning 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

35.  Aircraft  Waiting  for  Hand-off/Release 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

36.  Aircraft  Near  Exit  Fix/Arrival  Airport 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

37.  Density  of  Aircraft  on  Radar  Display 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

Please  circle  the  number  that  best  describes  the  importance  of  the  following  radar  display  information. 


38.  System  Clock 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

39.  VORs 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

40.  Fixes 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

~ 

10 

extremely 

high 

41.  Airports 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

~ 

10 

extremely 

high 

42.  Restricted  Area  Boundaries 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

43.1LS  Approaches 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

44.  Holding  Patterns 

extremely 

low 

I 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

45.  Obstructions 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

46.  Sector  Boundaries 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

47.  Filter  Settings 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

48.  Future  Aircraft  List 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

49.  Collision  Alert 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 
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APPENDIX  L 


Exit  Questionnaire 


Please  complete  the  following: 


1 .  Circle  the  number  that  best  describes  the 

Genera  Center  Airspace  hands-on  training. 

extremely  1  2 
poor 

3456789  10  extremely 
good 

2.  Circle  the  number  that  best  describes  the 

Decision  Support  Tool  hands-on  training. 

extremely  1  2 
poor 

3456789  10  extremely 
good 

3.  Was  there  anything  that  you  found  particularly  unique  in  the  simulation  that  you  would  not  see  at  your  home 
facility? 


4.  Were  you  constantly  aware  of  wearing  the  oculometer  or  did  you  tune  it  out? 


5.  How  do  you  decide  whether  or  not  to  suppress  data? 


6.  Is  there  anything  about  the  study  that  we  should  have  asked  or  that  you  would  like  to  comment  about? 


Thank  you  for  your  participation. 
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APPENDIX  M 


Biweekly  Schedule 


Monday 

Travel  to  the  FAA  WJH  Technical  Center 

Tuesday 

Introduction  to  the  experiment  and  first  day  of  DST  training:  Classroom 

Wednesday 

Second  day  of  DST  Training:  Classroom 

Thursday 

Third  day  of  DST  Training:  Hands-on 

Friday 

Airspace  training 

Monday 

Training  on  TGF/DSR/URET/SAVANT/SAGAT 

Tuesday 

Experimental  Simulations 

Wednesday 

Experimental  Simulations 

Thursday 

Experimental  Simulations 

Friday 

Travel  home 
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APPENDIX  N 


Statistical  Methods 

This  section  provides  background  information  on  the  statistical  methods  used  in  this  report. 
These  methods  are  powerful  tools  that  allow  researchers  to  determine  the  most  probable 
outcomes  of  an  experiment  based  on  limited  sample  sizes.  The  following  paragraphs  explain 
general  concepts  in  statistics,  the  methods  used  in  this  study  and  some  important  considerations 
in  order  to  use  them  effectively. 

The  purpose  of  any  statistical  experiment  is  to  determine  the  effect  of  certain  factors  on  one  or 
more  outcome  variables  (or  dependent  variables  [DVs]).  An  example  of  a  DV  is  the  number  of 
altitude  changes  an  ATCS  makes.  This  DV  could  be  affected  by  the  type  of  airspace  (terminal, 
en  route,  or  oceanic),  the  number  of  aircraft  flying  through  or  to  the  sector,  or  many  other 
factors.  The  manipulated  factors  of  an  experiment  are  the  independent  variables  (or  IVs). 

Each  manipulation  of  an  IV  (e.g.  25  planes  or  45  planes)  forms  a  separate  experimental 
condition.  Each  trial  under  a  particular  condition  is  termed  an  observation. 

Experiments  can  include  one  or  more  IVs.  When  an  experiment  includes.more  than  one  IV, 
multiple  IVs  can  affect  the  outcome  differently,  called  an  interaction.  It  would  be  impossible  to 
study  the  effects  of  type  of  airspace  and  number  of  aircraft  independently.  When  such 
interactions  between  IVs  occur,  the  researcher  will  study  the  effect  by  holding  one  variable 
constant  while  varying  the  others,  called  testing  for  simple  effects.  In  this  way,  the  researcher 
obtains  a  picture  of  how  the  variables  interact  by  examining  the  outcome  of  each  manipulation. 
When  researchers  study  the  effect  of  each  IV  separately  (no  interactions),  it  is  termed  an  analysis 
of  main  effects.  We  will  study  main  effects  only  in  the  absence  of  interactions. 

The  number  of  values  for  the  IVs  included  in  an  experiment  depends  on  several  practical 
considerations.  For  example,  if  a  researcher  is  studying  the  decision-making  patterns  of 
controllers  as  a  function  of  type  of  airspace,  the  values  of  the  IV,  type  of  airspace  (tower, 
TRACON,  en  route),  are  clear.  In  other  cases,  the  answer  would  depend  upon  what  type  of 
outcome  the  researchers  needed  from  the  results  of  the  experiment  as  well  as  some  practical 
considerations.  Different  values  of  IVs,  termed  levels,  can  increase  the  number  of  experimental 
conditions  and  thus  increase  the  resources  needed  to  complete  the  experiment.  One  can  imagine 
the  complexity  and  length  of  an  experiment  in  which  researchers  studied  controllers  with 
experience  ranging  from  1  to  50  years,  thus  creating  50  incremental  levels.  It  would  be  far 
simpler  and  easier  to  study  the  effect  of  controller  experience  by  using  only  three  categories: 
Developmental,  ATCS,  and  Supervisor. 

What  is  the  number  of  observations  required  for  each  test  condition?  Increasing  the  number  of 
observations  increases  the  statistical  power,  of  the  experiment.  Increased  statistical  power  means 
that  an  increased  probability  exists  that  the  outcome  of  an  experiment  will  likely  be  true  for  the 
entire  population.  However,  increasing  the  number  of  observations  comes  at  the  expense  of 
greater  numbers  of  participants,  more  time,  or  both.  An  efficient  experimental  design  should 
include  enough  observations  for  reasonable  statistical  power  without  including  unnecessary 
observations  that  could  dramatically  increase  demands  for  resources  unless  there  was  an 
increased  need  for  power. 
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With  insight  into  statistical  terminology  as  well  as  some  background  into  considerations  involved 
in  experimental  design,  it  is  now  useful  to  look  into  several  different  categories  of  experiments 
and  statistical  methods  used  to  determine  significant  outcomes.  For  simplicity,  each  of  the 
following  categories  involves  only  a  single  IV  (the  experience  level  of  controllers).  In  increasing 
level  of  complexity,  we  will  examine  three  categories  of  experiments: 

1 .  Observations  on  a  single  DV  under  two  conditions  (T-test) 

2.  Observations  on  a  single  DV  under  multiple  conditions  (ANOVA) 

3 .  Observations  on  multiple  DVs  under  multiple  conditions  (M ANOVA) 

We  will  discuss  each  of  these  categories  below. 

Observations  on  a  Single  Variable  under  Two  Conditions 

When  a  researcher  wants  to  compare  two  conditions,  he  or  she  takes  the  average  of  multiple 
observations  on  a  single  variable  under  two  conditions  and  performs  a  T-test.  However,  an 
average  value  can  often  be  misleading.  Within  a  group  of  such  observations,  some  differences 
will  exist  in  the  individual  observations  that  contributed  to  the  average.  Some  Developmental 
controllers  may  be  faster  learners  than  others  and  will  use  less  altitude  changes  in  order  to 
control  traffic.  The  average  number  of  altitude  changes  for  all  developmental  controllers  can 
include  a  wide  range  of  values.  The  differences  between  the  individual  times  and  the  mean 
number  of  times  represent  the  variability  of  the  data.  As  the  variability  in  the  data  increases,  the 
mean  value  is  less  useful  to  the  researcher  because  many  of  the  individual  values  are  far  from  the 
mean.  Figure  N-l  illustrates  the  variability  of  data. 
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Figure  N-l .  Two  sets  of  observations  with  the  same  means,  but  very  different  values. 

If  a  researcher  wants  to  compare  two  samples,  the  comparison  not  only  involves  comparing  the 
averages  but  also  the  variability  within  the  observations.  For  this  reason,  the  true  mean  (the 
mean  a  researcher  would  calculate  if  he  or  she  sampled  the  number  of  altitude  changes  for  all 
Developmental  controllers  in  the  world)  differs  from  the  sample  mean.  A  researcher  must  ask  if 
the  difference  in  the  means  of  these  two  sets  of  observations  is  a  true  difference,  or  caused  by 
chance.  This  is  where  probability  theory  aids  the  researcher.  Statistics  can  help  the  researcher 
determine  the  probability  that  the  two  means  for  the  entire  population  (all  controllers)  are 
different  from  the  sample  (limited  number  of  controllers).  The  statistical  test  used  in  this  case  is 
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the  /-test.  The  t-test  compares  two  averages  and  checks  if  the  two  averages  are  different  due  to 
chance  alone.  It  is  important  to  recognize  that  the  t-test  never  gives  the  researcher  100% 
assurance  that  the  two  means  actually  differ.  It  is  common  practice  to  accept  a  95%  assurance 
(or  in  other  words  a  5%  risk)  as  sufficient  guarantee. 

SUMMARY  OF  A  T-TEST.  An  experiment  includes  multiple  observations  on  a  single  variable 
under  two  conditions.  The  average  values  (means)  of  the  two  conditions  take  variability  into 
consideration.  The  analyses  determine  the  probability  that  the  means  differ  due  to  chance  alone. 

Example:  When  one  compares  the  number  of  altitude  changes  between  Developmental  and 
ATCS  controllers  at  a  local  center,  the  comparison  involves  multiple  observations.  The  multiple 
observations  consist  of  the  number  of  altitude  changes  of  each  individual  within  each  experience 
level.  The  variable  is  the  number  of  altitude  changes.  The  conditions  include  the  two  levels  of 
experience.  Figure  N-2  is  a  graphical  display  of  this  example.  Although  it  shows  a  difference  in 
number  of  altitude  changes  between  the  two  groups,  some  individual  observations  overlap.  The 
researcher  uses  a  t-test  to  see  if  chance  caused  this  difference. 
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Figure  N-2.  Multiple  observations  of  altitude  changes  as  a  function  of  experience  level. 

Observations  on  a  Single  Variable  under  Multiple  Conditions 

Where  the  t-test  compared  the  averages  between  two  conditions,  the  analysis  of  variance 
(ANOVA)  compares  averages  of  a  single  variable  between  multiple  conditions  (i.e.  the  number 
of  altitude  changes  including  Developmental,  ATCS,  and  Supervisors).  An  ANOVA  tests  if 
these  averages  are  different  due  to  chance  alone.  The  basic  test  results  in  an  F  value  for  a  single 
DV  (the  number  of  altitude  changes).  The  value  of  F  ranges  from  0  to  infinity  (oo).  A  large  F 
value  may  indicate  that  the  IV  (experience  level)  has  a  powerful  effect  on  the  DV  (number  of 
altitude  changes)  with  less  likelihood  that  differences  between  means  occurred  by  chance.  The 
strength  of  association  (e.g.  r|)  or  percent  of  variance  explained  is  an  indication  of  the  difference 
in  the  strength  of  effects  between  conditions.  A  difference  between  means  is  significant  if  there 
is  a  very  high  probability  that  the  means  are  actually  different  (usually  greater  than  95%). 

Sometimes,  there  is  a  significant  difference  where  the  F  value  is  relatively  low.  This  indicates 
that  the  IV  does  not  have  a  very  strong  effect. 
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An  ANOVA  can  show  that  there  is  a  difference  in  means  not  caused  by  chance  alone.  If  the 
ANOVA  indicated  that  the  number  of  altitude  changes  varies  with  experience  level,  is  the  mean 
number  of  altitude  changes  for  Developmental  controllers  different  than  ATCS  controllers?  The 
mean  for  Developmental  controllers  differs  significantly  from  those  of  ATCS  but  not 
significantly  from  those  for  Supervisors.  Therefore,  the  researcher  runs  another  test  to 
compliment  the  ANOVA,  called  a  post-hoc  comparison.  Researchers  will  use  post-hoc 
comparisons  to  determine  which  of  the  pairs  of  means  differ  significantly. 

SUMMARY  OF  AN  ANOVA.  The  analysis  of  variance  (ANOVA)  compares  averages  of  a 
single  DV  between  multiple  conditions  and  tests  if  these  averages  ar.e  different  due  to  chance 
alone.  The  test  results  in  an  F  value.  A  large  F  value  indicates  less  likelihood  and  a  small  value 
indicates  increased  likelihood  that  differences  between  means  occurred  by  chance.  A  difference 
between  means  is  significant  if  there  is  a  very  high  probability  that  the  means  are  actually 
different.  A  post-hoc  comparison  determines  which  means  differ. 

Example:  When  a  researcher  compares  the  number  of  altitude  changes  between  Developmental, 
ATCS  and  Supervisors  at  a  local  center,  the  comparison  involves  multiple  observations.  The 
multiple  observations  are  the  number  of  altitude  changes  of  each  individual  within  each  group. 
The  variable  is  the  number  of  altitude  changes.  The  conditions  are  the  three  experience  levels. 
Figure  N-3  displays  the  data  related  to  this  example.  Some  differences  in  number  of  altitude 
changes  exist  between  experience  levels,  but  there  is  overlap  between  observations  in  each 
experience  level.  An  ANOVA  would  determine  if  these  differences  were  due  to  chance  alone.  If 
the  ANOVA  indicated  that  there  is  some  difference  in  experience  levels  regarding  number  of 
altitude  changes,  post-hoc  comparisons  would  indicate  which  means  associated  with  which 
experience  levels  differ. 


Figure  N-3.  Multiple  observations  of  altitude  changes  as  a  function  of  experience  level. 

Observations  on  a  Multiple  Variable  under  Multiple  Conditions 

Where  an  ANOVA  compares  averages  between  multiple  conditions  for  a  single  variable  (a 
univariate  test),  the  multivariate  ANOVA  (MANOVA)  compares  averages  for  several  DVs 
simultaneously  and  tests  if  these  averages  are  different  due  to  chance  alone.  Suppose  that 
researchers  wanted  to  examine  number  of  altitude  changes  and  the  number  of  heading  changes. 
Also,  suppose  that  the  researchers  wanted  to  test  both  of  these  measures  as  a  function  of 
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experience  level.  This  presents  a  case  of  multiple  DVs  (number  of  altitude  changes  and  number 
of  heading  changes)  and  multiple  conditions  (Developmental,  ATCS  and  Supervisor).  The 
experimenters  focus  on  how  experience  level  affects  the  set  of  variables  (number  of  altitude 
changes  and  number  of  heading  changes).  A  researcher  would  not  do  two  ANOVAs  for  each  of 
the  DVs  (number  of  altitude  changes  and  number  of  heading  changes)  because  misleading 
outcomes  result  from  multiple  ANOVAs.  A  MANOVA  is  more  appropriate. 

The  basic  MANOVA  test  results  in  a  value  called  Wilk’s  Lambda  (A)  that  includes  the  effects 
of  more  than  one  DV  (both  number  of  altitude  changes  and  number  of  heading  changes).  The 
value  of  Wilk’s  Lambda  ranges  from  zero  to  one.  The  lower  the  value  of  A,  the  more  powerful 
the  effect  of  the  IV  (experience  level)  on  the  set  of  DVs  and  the  less  likely  it  is  that  the 
differences  between  means  occurred  by  chance.  Sometimes,  there  is  a  significant  difference 
where  Wilk’s  Lambda  is  relatively  high.  This  indicates  that  the  effect  is  not  that  strong. 

After  a  significant  result  of  a  MANOVA  test,  researchers  then  conduct  ANOVA  tests  (one  for 
number  of  altitude  changes  and  one  for  number  of  heading  changes).  Figure  N-4  depicts  an 
example  of  the  steps  taken  during  a  MANOVA.  The  example  shown  in  Figure  N-4  includes  two 
DVs. 


Figure  N-4.  Example  of  the  steps  in  a  Multivariate  Analysis  of  Variance  (MANOVA). 

SUMMARY  OF  A  MANOVA.  The  multivariate  ANOVA  (MANOVA)  compares  averages  for 
several  variables  simultaneously  and  tests  if  these  averages  are  different  due  to  chance  alone. 

The  basic  MANOVA  results  in  a  value  called  Wilk’s  Lambda  (A)  that  includes  the  effects  of 
more  than  one  DV.  The  lower  the  value  of  A,  the  more  powerful  the  effect  of  the  IV  on  the  set 
of  DVs  and  the  less  likely  it  is  that  the  differences  between  means  occurred  by  chance.  After  a 
significant  result  of  a  MANOVA  that  indicates  that  at  least  two  means  are  statistically  different 
for  the  system,  researchers  then  conduct  ANOVAs. 
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Example:  When  one  compares  the  number  of  altitude  changes  and  number  of  heading  changes 
between  Developmental,  ATCS  and  Supervisor  at  a  local  center,  the  comparison  involves 
multiple  observations  of  two  variables.  The  multiple  observations  are  the  number  of  altitude 
changes  and  number  of  heading  changes  of  each  individual  within  each  experience  level.  The 
DVs  are  the  number  of  altitude  changes  and  the  number  of  heading  changes.  The  three 
experience  levels  form  the  conditions.  Figure  N-5  displays  the  data  for  this  example.  Without 
looking  at  the  individual  variables,  one  can  see  that  the  three  experience  levels  differ.  A 
MANOVA  would  determine  if  chance  alone  caused  these  differences.  If  the  differences  are 
beyond  chance  (or  are  significant  in  statistical  terms),  the  researcher  then  runs  ANOVAs  on  the 
individual  variables. 
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Figure  N-5.  Multiple  observations  of  number  of  altitude  changes  and  number  of  heading 

changes  as  a  function  of  experience  level. 

Summary 

The  preceding  paragraphs  give  some  insight  into  the  statistical  methods  used  by  researchers. 
Statistical  methods  are  very  powerful  tools  for  the  researcher.  They  tell  the  researcher  if  the 
experimental  conditions  affect  the  dependent  measures  tested.  The  type  of  statistical  test  that  the 
researcher  uses  varies  with  the  type  of  experiment.  A  good  researcher  will  design  experiments 
so  they  can  use  these  techniques  fully. 


N-6 


APPENDIX  O 


SAVANT  SA  Results 


Table  0-1.  Current  and  Future  S A  (Correct  Responses  Only):  Means  and  Standard 

Deviations  (seconds). 


Question 

£ 

mmaifflffl 

Mr..: 

e  Controller  Pc 

iiisi 

Positimi  i 

Load/Pos. 

Collapsed 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

E2£ 1 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Current  RT 

4.88 

1.16 

4.22 

1.22 

4.55 

1.22 

4.74 

1.26 

m 

5.00 

1.17 

4.81 

1.19 

4.74 

1.24 

4.78 

1.21 

vutoi 

■g&Ib 

Future  RT 

5.19 

1.04 

5.63 

1.02 

5.41 

1.04 

5.18 

0.82 

11 

5.55 

0.93 

5.18 

0.92 

5.68 

0.99 

5.43 

Current  RT 

4.87 

1.17 

4.91 

1.07 

4.89 

1.10 

4.76 

0.73 

5.20 

1.69 

4.98 

1.30 

4.81 

0.96 

5.06 

1.40 

4.94 

1.20 

Future  RT 

4.95 

0.98 

5.40 

0.88 

5.24 

0.93 

5.47 

1.26 

5.56 

0.78 

5.51 

1.04 

5.21 

1.15 

5.46 

0.84 

5.35 

0.98 

Current  RT 

4.72 

0.93 

4.49 

0.71 

4.61 

0.82 

4.74 

0.66 

4.84 

1.27 

4.79 

0.99 

4.73 

0.79 

4.66 

1.02 

4.70 

0.91 

Aut&fc 

Future  RT 

4.89 

1.15 

7.61 

2.07 

5.32 

1.62 

5.01 

1.51 

5.14 

1.03 

5.07 

1.27 

4.95 

1.32 

5.53 

1.49 

5.17 

1.40 

Current  RT 

4.82 

1.07 

4.54 

1.04 

4,68 

1.06 

4.75 

0.90 

5.10 

1.35 

4.92 

1.15 

4.78 

0.99 

4.82 

1.23 

4.80 

1.11 

CflH. 

Future  RT 

5.01 

1.04 

5.61 

1.12 

5.31 

1.12 

5.22 

1.22 

5.48 

0.95 

5.35 

1.10 

5.11 

1.13 

5.55 

1.04 

5.33 

1.11 

Table  0-2.  Current  and  Future  SA  (All  Responses):  Means  and  Standard  Deviations 

(seconds). 


mmm 

»« 

icsui 

gggyK 

2*  iM 

i;::  ..  1 

Eli 

saifflfflii 

■ALV 

as® 

l||| 

Load/Pos. 

Low  Load 

High  Load 

■ 

ESS 

W»M 

muil 

laiMl 

Ml 

waslm 

EuS 

msa 

0.81 

5.00 

■ 

0.71 

5.19 

1.09 

4.97 

1.10 

5.27 

0.96 

5.10 

' 

Current  Correct 

58.40 

30.20 

32.20 

63.80 

31.20 

54.36 

37.40 

58.60 

19.20 

56.40 

29.30 

56.30 

33.50 

63.90 

26.67 

30.30 

Future  RT 

5.27 

1.04 

1.07 

5.53 

1.07 

5.21 

0.71 

5.78 

0.63 

5.49 

0.72 

5.24 

0.88 

5.78 

5.51 

0.90 

81.10 

19.70 

18.60 

78.10 

19.10 

79.30 

13.40 

67.80 

24.80 

73.60 

20.40 

80.20 

16.60 

71.53 

21.90 

75.90 

19.80 

5.14 

1.33 

5.29 

1.14 

5.21 

1.22 

5.35 

1.49 

6.00 

1.30 

5.68 

1.41 

5.25 

1.39 

5.65 

1.25 

5.45 

1.33 

69.80 

20.90 

66.10 

31.40 

68.00 

26.30 

54.99 

37.64 

63.70 

36.70 

59.30 

36.80 

62.40 

30.90 

64.90 

33.60 

63.60 

32.00 

Auto  2 

5.02 

0.62 

5.49 

1.15 

5.25 

0.94 

5.37 

1.32 

5.49 

0.85 

5.43 

1.10 

5.19 

1.03 

5.49 

1.00 

5.34 

1.01 

82.20 

21.20 

77.30 

15.20 

79.70 

18.30 

78.00 

23.80 

79.00 

18.10 

78.40 

20.80 

80.10 

22.30 

78.15 

16.40 

79.10 

19.40 

Current  RT 

4.93 

0.87 

4.91 

0.55 

4.92 

0.18 

4.99 

0.68 

4.99 

1.40 

4.99 

1.08 

4.96 

0.77 

4.95 

1.05 

4.95 

0.91 

Current  Correct 

52.50 

33.60 

76.20 

22.80 

64.40 

30.70 

48.80 

37.30 

66.40 

37.80 

57.60 

38.00 

50.70 

35.00 

71.30 

31.10 

61.00 

34.40 

Future  RT 

4.97 

1.02 

5.63 

1.06 

5.30 

1.08 

5.26 

0.92 

1.00 

0.76 

5.18 

0.95 

5.11 

0.97 

5.36 

1.05 

5.24 

1.02 

PpjSl]  Future  Correct 

77.60 

28.10 

77.30 

19.50 

77.40 

23.80 

76.90 

24.40 

76.20 

20.50 

76.50 

22.20 

77.30 

25.90 

76.70 

19.70 

77.30 

22.80 

JSSl]  Current  RT 

5.01 

1.01 

5.08 

0.98 

5.05 

0.99 

5.10 

1.21 

5.47 

1.22 

5.29 

1.22 

5.06 

1.11 

5.27 

1.12 

5.17 

1.12 

■SHM  Current  Correct 

60.30 

29.00 

70.50 

28.80 

65.40 

29.20 

52.70 

36.70 

62.90 

31.90 

57.80 

34.60 

56.50 

33.10 

66.70 

30.50 

61.60 

32.20 

fflHFutiiie  RT 

5.08 

0.90 

5.64 

1.08 

5.36 

1.03 

5.28 

1.00 

5.45 

0.87 

5.37 

0.94 

5.18 

0.95 

5.54 

0.98 

5.36 

0.98 

Future  Correct 

80.30 

22.90 

76.60 

17.50 

78.40 

20.38 

78.11 

20.78 

74.30 

21.40 

76.20 

21.00 

79.20 

21.70 

75.40 

19.50 

77.30 

20.70 

Table  0-3.  SAGAT  type  SA:  Means  and  Standard  Deviations  (percent)  (N=16). 


N=16 

Question 

Low  Load 

c  Controller  ft 

Hill 

Low  Load 

Highroad 

Load  Coll. 

Low  Load 

Load/Pos. 

Collapsed 

mil 

USUI 

mil 

EBll 

Means  j 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

•v-  j  ■*  | 

Percent  Answered 

105.40 

23.50 

105.30 

35.00 

105.40 

23.00 

106.60 

21.80 

87.80 

31.10 

97.20 

28.10 

106.00 

22.30 

96.60 

33.80 

101.30 

28.80 

Autol 

Percent  Correct 

34.70 

17.90 

37.50 

18.80 

36.10 

18.10 

22.50 

16.70 

27.80 

16.50 

25.20 

16.50 

28.60 

■uni 

32.60 

18.10 

30.60 

18.10 

H| 

Percent  Answered 

102.50 

29.00 

88.90 

25.30 

95.70 

27.70 

102.70 

32.30 

111.90 

36.30 

107.30 

34.10 

102.60 

30.20 

100.40 

32.90 

101.50 

31.40 

AutoZ 

Percent  Correct 

33.90 

13.30 

32.30 

10.20 

33.10 

11.70 

28.10 

15.00 

25.50 

15.60 

26.80 

15.10 

31.00 

14.20 

28.90 

13.50 

30.00 

13.80 

l  ■■  ■  j' >; 

Percent  Answered 

89.50 

45.70 

97.60 

36.50 

93.90 

51.40 

92.60 

53.40 

93.20 

51.60 

99.80 

39.60 

91.00 

48.00 

95.40 

44.40 

mMm 

Percent  Correct 

24.60 

17.40 

26.30 

16.00 

16.40 

14.20 

27.00 

18.30 

21.70 

17.00 

22.20 

15.50 

25.80 

17.60 

24.00 

16.50 

mm 

Percent  Answered 

104.50 

24.80 

94.60 

36.30 

99.60 

31.30 

101.10 

36.80 

97.40 

41.90 

99.20 

39.30 

102.80 

31.30 

96.00 

39.00 

99.40 

35.40 

Cull 

Percent  Correct 

32.20 

15.50 

31.50 

16.50 

31.90 

15.90 

22.40 

15.70 

26.80 

16.50 

24.60 

16.20 

27.30 

16.30 

29.10 

16.60 

28.20 

16.40 

0-1 


Table  0-4.  SAGAT  type  SA:  Means  and  Standard  Deviations  (percent)  (N=6). 


N=6 


Question 


Low  Load 


Means  SDs 


High  Load 


Means  [  SDs 


Load  Coll. 


Means  I  SDs 


Low  Load 


Means  SDs 


High  Load 


Means  |  SDs 


Load  Coll. 


Means  SDs 


Low  Load 


Means  SDs 


High  Load 


Means  |  SDs 


Load/Pos. 

Collapsed 


Means  SDs 


Percent  Answered 
Percent  Correct 


120.90 

41.70 


23.40 

16.50 


127.20 

53.90 


40.70 

20.30 


124.00 

47.80 


3  3.80 
18.70 


123.60 

26.10 


20.50 

14.60 


103.30 

27.70 


21.00 

19.50 


113.40 

26.90 


22.40 

16.40 


122.20 

33.90 


21.00 

17.00 


115.20 

40.80 


33.30 

23.40 


118.70 

37.30 


27.50 

20.30 


Percent  Answered 
Percent  Correct 


112.90 

37.70 


34.60 

7.40 


100.90 

27.60 


32.60 

5.10 


106.90 

32.70 


32.60 

8.10 


99.50 

25.00 


35.40| 

17.80 


122.80 

35.90 


42.60| 

11.10 


111.20 

30.50 


39.30 

15.30 


106.20 

31.40 


34.10| 

14.60 


111.80 

31.70 


37.90 

9.30 


109.00 

3.16 


35.40 

12.00 


Percent  Answered 
Percent  Correct 


119.00 

34.90 


29.30 

18.90 


108.40 

33.70 


21.70 

13.10 


113.70 

34.30 


25.20 

15.50 


101.30 

14.90 


75.00 

16.80 


117.40 

33.50 


24.20 

8.70i 


109.40 

24.20 


53.80 

16.00 


110.10 

24.90 


55.10 

20.00' 


112.90 

33.60 


22.40 

10.60 


111.50 

29.20 


41.10 

16.30 


Percent  Answered! 
Percent  Correct 


117.60 

38.10 


27.90 

14.50 


112.20 

38.40 


32.70 

17.70 


114.90 

38.20 


30.00 

15.90 


108.10  47.70 
22.10  16.30 


114.50 

32.40 


30.10 

13.50 


111.30 

27.20 


39.40 

15.70 


1 12.90 
30.00 


38.80 

17.30 


113.30 

35.40 


3l.00| 

15.80 


113.10 

32.70 


34.90 

16.60 


Table  0-5.  Current  S A  (Correct  Responses  Only):  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-level 

Position 

1 

15 

.029 

Load 

1 

m 

.925 

Automation 

1.014 

m 

■efl! 

.402 

Position  X  Load 

4.964 

HESfc 

1 

15 

— tEH 

Position  X  Automation 

0.618 

MESsIS 

«T 

« 

Load  X  Automation 

WKKEBKL 

mm 

Position  X  Load  X  Automation 

0.760 

1.038 

0.732 

m 

1! 

Table  0-6.  Future  SA  (Correct  Responses  Only):  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

KTH 

/?-level 

Position 

■keee 

HHESfiS 

15 

—hill 

Load 

8.924 

■K HE 

1 

15 

mgm 

1.032 

m 

.401 

Position  X  Load 

HU 

i 

15 

IHI 

Position  X  Automation 

WIKI 

m 

Hi 

Load  X  Automation 

0.144 

1.021 

0.141 

.869 

Position  X  Load  X  Automation 

0.689 

1.353 

0.509 

m 

■Hi! 

.606 

Table  0-7.  Current  SA  (All  Responses):  MANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy’s 

F 

df  1 

df  2 

p-level 

Position 

.610 

0.390 

0.639 

0.639 

4.476 

2 

14 

.031 

Load 

.628 

0.372 

QB2 

m 

HI 

.038 

Automation 

—Ml 

m 

H 

.216 

Position  X  Load 

m 

H 

.728 

Position  X  Automation 

m 

12 

.822 

Load  X  Automation 

0.361 

m 

m 

— na 

Position  X  Load  X  Automation 

0.159 

0.189 

0.189 

IH 

m 

HID 

0-2 


Table  0-8.  Response  Time  for  Current  SA  (All  Responses):  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df2 

p-level 

Position 

2.800 

0.507 

5.520 

1 

15 

.033 

Load 

1.328 

1.693 

1 

15 

.213 

Automation 

mg 

1.028 

3.977 

2 

m 

.029 

Position  X  Load 

0.532 

m 

.477 

Position  X  Automation 

0.635 

30 

.523 

Load  X  Automation  * 

0.685 

1.431 

0.479 

2 

30 

.624 

Position  X  Load  X  Automation 

0.239 

0.980 

0.243 

2 

30 

.785 

Table  0-9.  Percentage  Correct  for  Current  SA  (All  Responses):  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/7-level 

Position 

0.277 

0.076 

3.622 

1 

15 

.076 

Load 

0.502 

mEfl£& 

1 

15 

.060 

Automation 

0.022 

mum 

mam 

30 

.849 

Position  X  Load 

0.000 

0.000 

2 

30 

.998 

Position  X  Automation 

0.001 

.980 

Load  X  Automation 

0.140 

.299 

Position  X  Load  X  Automation 

0.092 

0.513 

2 

30 

.604 

Table  0-10.  Future  SA  (All  Responses):  MANOVA  Results. 


Hotelling 

Roy's 

F 

df  1 

df  2 

p-level 

Position 

WEEM 

KflEBgl 

K 

.638 

Load 

1 

m 

Automation 

.778 

0.222 

4 

12 

.517 

Position  X  Load 

.845 

0.155 

0.183 

0.183 

1.282 

2 

m 

Position  X  Automation 

.899 

0.101 

0.113 

0.113 

0.338 

4 

m 

.847 

Load  X  Automation 

0.160 

0.479 

4 

.751 

m 

4 

m 

.751 

Table  0-11.  Response  Time  for  Future  SA  (All  Responses):  ANOVA  Results. 


Position _ 

Load 

Automation _ 

Position  X  Load 
Position  X  Automation 
Load  X  Automation 
Position  X  Load  X  Automation 


MS  Effect 
0.000 
6.078 
1.104 
1.600 
0.440 
0.456 
0.690 


MS  Error 
089' 
0.803' 
0.841' 
0.660' 
0.632' 
0.707' 
1.107' 


F 

0.000' 

7.570' 

1.313' 

2.427' 

0.696' 

0.645' 

0.623' 


df  1 


df  2 


15 


15 


3d 


3d 


3d 


3d 


30 


/7-level 

1983 

.015 

.284 

.140 

.506 

.532 

.543 


Table  0-12.  Percentage  Future  for  Current  SA  (All  Responses):  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/7-level 

Position 

0.043 

0.043 

1 

m 

.336 

Load 

0.097 

0.036 

m 

m 

.120 

Automation 

0.018 

0.052 

m 

■Ei! 

.713 

Position  X  Load 

■msog 

■HSS! 

30 

.834 

Position  X  Automation 

imsss 

mmm 

m 

30 

.917 

Load  X  Automation 

mm 

mmm 

m 

30 

.517 

Position  X  Load  X  Automation 

0.015 

0.029 

30 

.593 

0-3 


Table  0-13.  SAG  AT  type  Queries:  MANOVA  Results  (N=l  6). 


Position _ 

Load 


mtomation 
Position  X  Load 


osition  X  Automation 


Load  X  Automation 


Position  X  Load  X  Automation 


Wilks  A  Pillai's  Hotelling  Roy’s  F  df  1  df  2 
.446  "0.554  1.242  1.242  8.696  2 ~A 

.694  0.306  0.440  0.440  3.080 


.637  0.363  0.569  0.569  1.708 

.943  0.057  0.060  0.060  “Ch42l|  7 


1  .741  0.259  0.35 


p-  lev  1 


Table  0-14.  Percent  Questions  Answered:  ANOVA  Results  (N=  16). 


[Position 


Load 


utomation 


osition  X  Load 


Position  X  Automation 


MS  Effect  MS  Error 
6500  0432 


F  df  1  df  2  |/?-level 
0.004  r  15 


Load  X  Automation 

0.0251  0.0851 

0.298 

Position  X  Load  X  Automation 

o 

3 

O 

1.667 

Table  0-15.  Percent  Correct:  ANOVA  Results  (N=l 6). 


Position  X  Load  X  Automation 


0221  0.9781  43.6731  43.6731  21.836: 


Table  0-17.  Percent  Questions  Answered:  ANOVA  Results  (N=6). 


p-level 


<U  o 


Table  0-18.  Percent  Correct:  ANOVA  Results  (N=6). 


MS  Effect 

MS  Error 

HBBH 

£31 

E29 

Position 

0.220 

0.013 

i 

E 

aaftiE 

No  Automation 

0.525 

0.006 

1 

5 

Limited  Automation 

MBEM! 

Full  Automation 

Load 

0.051 

0.013 

■ 

m 

Automation 

0.011 

E 

HS 

D-Side 

WKEQ 

m 

m 

■ggg- 

R-side 

0.165 

m 

m 

HIE 

Position  x  Load 

■n 

HKESi 

1 

5 

.388 

Position  x  Automation 

MKK; 

10 

.017 

Load  x  Automation 

0.011 

10 

.630 

Position  x  Load  x  Automation 

0.048 

0.026 

1.861 

2 

10 

.206 

0-5 


APPENDIX P 


SAGAT  SA  Results 


Table  P-1.  Level  1  SA  Items:  Means  and  Standard  Deviations  (percent). 


Level  1 


Question 


Low  Load 


Means  SDs 


High  Load 


Means  )  SDs 


Load  Coll. 


Means  |  SP7 


-  >y;:  u 


Low  Load 


Means  SDs 


High  Load 


Means  SDs 


Load  Coll. 


Means  SDs 


Low  Load 


Means  SDs 


High  Load 


Means)  SDs 


Load/Pos. 

Collapsed 


Means )  SDs" 


Aircraft  Position 
Callsign  Letter 
Callsign  Number 
Altitude 
Speed 
Heading 
Vertical  Change 
Turning 
Aircraft  Type 
Level  of  Control 


Aircraft  Position 
Callsign  Letter 
Callsign  Number 
Altitude 
Speed 
Heading 
Vertical  Change 
Turning 
Aircraft  Type 
Level  of  Control 


Aircraft  Position 
Callsign  Letter 
Callsign  Number 
Altitude 
Speed 
Heading 
Vertical  Change 
Turning 
Aircraft  Type 
Level  of  Control 


Aircraft  Position 
Callsign  Letter 
Callsign  Number 
Altitude 
Speed 
Heading 
Vertical  Change 
Turning 
Aircraft  Type 
Level  of  Control 


25.68 

30.95 

0.20 

27.77 
0.79 

64.42 

83.92 

95.51 

15.78 
0.60 


9.56 

39.76 

0.28 

33.03 

0.77 

46.54 

36.69 

32.44 

24.50 

1.13 


27.23  8.85 
11.81  24.81 
0.20  0.28 
18.22  24.94 
0.79  0.77 
56.61  45.29 
82.93  41.24 
98.51  24.75 
15.78  24.50] 
3.96  4.53 


26.46  9.10 
21.48  33.22| 
0.20  0.27 
23.02  28.97 
0.79  0.75 
60.59  45.28 
83.43  38.40] 
97.21  28.68 
15.78  24 JO 
2.28  3.67 


23.20 

21.40 
0.20 

35.40 
0.40 

78.10 

87.10 

97.10 
12.70 

10.40 


13.07 

33.76 

0.28 

38.43 

0.72 

46.30 

39.33 

24.87 
24  .*83 

24.88 


27.60 
2.00 
0.20 

26.10 

0.70 

56.60 

87.10 

97.10 
14.70 

3.90 


8.95 

2.76 

0.28 

33.60 

0.77 

45.29 

43.14 

33.59 

24.62 

4.53 


25.47 

11.81 

0.20 

30.84 

0.63 

68.10 

87.13 

97.11 

13.79 

7.18 


11.25] 

24.41 

0.27 

35.76 

0.73 

46.05 

40.65 
29.20 
*24.33 

17.66 


24.46 

26.25 

0.20 

31.64 

0.63 

71.61 

85.57 

96.35 

14.29 

5.51 


11.34 

36.43 

0.27 

35.39 

0.73 

46.10] 

37.61 
28.80 
24.28 

17.62 


27.45 

6.92 

0.20 

22.19 

0.79 

56.61 

85.10 

97.87 

15.28 

3.96 


8.76 

17.65 

0.27 

29.18 

0.75 

44.56 

41.59 

29.11 

24.16 

4.46 


25.96  10.17 
16.67  29.08 
0.20  0.27 
26.95  32.39 
0.71  0.74 
64.42  45.41 
85.34  39.36] 
97.16  28.73 
14.79  24.03 
4.73  12.75 


30.79 

11.81 

0.20 

29.40 

0.79 

56.61 

74.44 

98.24 

6.03 

1.12 


15.18 

24.81 

0.28 

32.68 

0.77 

45.29 

41.11 

24.60 

5.12 

2.24 


26.54 

13.14 

0.34 

13.56 

1.41 

46.13 

68.85 

97.64 

26.35 

3.46 


7.30 

24.64 

0.33 

9.02 

0.82 

38.46 

37.16 

24.17 
32.78 

4.39 


28.68 

12.47 
0.27 

21.55 

1.10 

51.47 
71.70 
97.95 
16.28 

2.29 


11.88 

24.33 

0.30 

24.16j 

0.81 

41.52 

38.63 

24.01 

24.03 

3.62 


23.50 
11.80 

0.20 

37.00 

0.90 

39.40 

79.50 
98.70 

5.00 

1.10 


9.83 

24.81 

0.28 

38.43 

0.77 

39.66 

44.10 

24.87 

4.90 

2.24 


19.60 
3.30 
0.30 

38.60 
0.90 

54.60 
77.00 
93.20 
51.50 

3.40 


8.31 

3.28 

0.33 

37.94 

0.80 

41.20 

42.71 

38.62 

45.85 

4.39 


21.63 

7.59 

0.27 

37.84 

0.94 

47.20 

78.31 

96.55 

29.14 

2.29 


9.16 

17.62| 

0.30 

37.57| 

0.77 

40.17 

42.72 

32.67 

35.79 

3.62 


27.20 

11.81 

0.20 

33.25 
0.86 

48.26 
77.05 
98.51 

5.52 

1.12 


13.05 

24.41 

0.27 

35.19 

0.75 

42.36 

42.02 

24.35 

4.94 

2.20 


23.13 

8.26 

0.34 

26.32 

1.18 

50.43 

73.08 

95.73 

39.30 

3.46 


8.44 

17.58 

0.32 

28.40 

0.81 

39.33 

39.57 

32.08 

40.20 

4.32 


25.17 

10.04 

0.27 

29.80 

1.02 

49.35 

75.10 

97.30 

22.76 

2.29 


11.081 

21.13 

0.30| 

31.81 

0.79 

40.56 

40.53 
28.51 

30.53 
3.59| 


25.31 
3.36 

10.14 

31.02 

10.58 

27.93 

84.89 

97.31 
16.77 

1.52 


10.22 

3.28 

24.95 

31.86 

24.88 

28.16 

31.44 

23.90 

24.35 

2.27 


26.25 
2.69 
0.27 

27.77 

10.74 

67.21 

88.01 

93.26 
13.79 

3.66 


7.70 

3.06 

0.31 

33.03 

24.85 

45.06 

38.65 

38.62 

24.74 

4.33 


25.78 

3.03 

5.21 

29.40 

10.66 

48.82 

86.49 

95.51 

15.28 

2.59 


8.92 

3.13 

17.65 

31.97 

24.46 

39.77 

34.71 

31.92 

24.16 

3.57 


20.10 

13.10 

0.30 

8.40 

0.70 

37.20 

82.90 

98.50 

6.00 

1.50 


8.63 

24.64 

0.33 

8.36 

0.77 

34.32 

41.24 

24.75 

5.12 

2.27 


22.10 

2.60 

0.20 

27.70 
1.00 

74.10 

56.70 
97.90 

5.00 

3.60 


10.15 

3.06 

0.31 

33.03 

0.82 

45.05 

37.49 

24.41 

4.90 

4.33 


21.15 

7.93 

0.30 

18.22 

0.94 

57.14 

71.07 

98.24 

5.52 

2.59 


9.31 

17.60| 

0.31 

24.54] 

0.79 

42.04] 

40.61 

24.20 

4.94 

3.57 


22.75 

8.26 

5.24 

19.89 

5.69 

32.61 

83.92 

97.95 

11.42 

1.52 


9.66 

17.58 

17.64 

24.27 

17.60 

31.05 

36.09 

24.01 

17.66] 

2.23 


24.19 

2.69 

0.27 

27.77 

5.93 

70.76 

74.44 

95.94 

9.42 

3.66 


9.11 

3.01 

0.30 

32.49 

17.58 

44.44 

40.44 
32.24 
17.77 

4.26 


23.47 

5.48 

2.76 

23.85 

5.81 

53.04 

79.42 
97.03 

10.42 
2.59 


9.34 

12.64 

12.48 

28.55 

17.45 

40.72 

38.36 

28.33 

17.58 

3.54 


27.27 

15.49 

3.52 

29.40 
4.06 

50.41 
81.33 
97.13 
12.87 

1.08 


11.91 

27.60] 

14.41 

31.86] 

14.38 

41.59 

36.20 

26.84 

19.97 

1.94 


26.68 

9.22 

0.27 

19.89 

4.32 

56.97 

80.62 

96.81 

18.67 

3.70 


7.82 

20.06] 

0.30 

24.24 

14.36 

42.67 

39.14 

29.79 

27.26 

4.33 


26.97 
12.36 

1.89 

24.67 

4.19 

53.73 

80.98 
96.97 
15.78 

2.39 


10.02 

24.08 

10J9 

28.36 

14.29 

41.99 

37.50 

28.21 

23;84| 

3.59 


22.33 
15.49 

0.25 

27.22 

0.73 

53.21 

83.34 
98.19 

7.95 

4.35 


10.55 

27.60 

0.29 

32.32 

0.73 

42.37] 

41.05 

24.30 

14.76 

14.42 


23.17 

2.69 
0.27 

30.90 

0.94 

62.22 

74.92 

96.35 

24.41 

3.70 


9.58 

3.00| 

0.30] 

34.42 

0.78 

43.59 

42.05 

32.36 

32.53 

4.33 


22.75 

9.11 

0.26 

29.07 

0.84 

57.80 

79.32 

97.35 

16.24 

4.02 


10.03 

19.96 

0.29 

33.22 

0.76 

42.90 

41.64! 

28.68 

25.69 

10.59 


24.81 
15.49 

1.88 

28.31 

2.40 

51.81 
82.35 
97.69 
10.42 

2.72 


11.45 

27.46 
10.19 
31.93 
10.18 
41.78 

38.53 
25.58 

17.54 
10.29 


24.93 

5.96 

0.27 

25.43 

2.63 

59.62 

77.85 

96.59 

21.55 

3.70 


8.88 

14.42 

0.30 

29.81 

10.17 

42.95 
40.51 

30.95 
29.91 

4.31 


24.87 
10.74 

1.08 

26.88 
2.51 

55.78 

80.16 

97.16 

16.01 

3.21 


10.22 

22.08 

7.21 

30.83 

10.15 

42.37 
39.53 

28.38 
24.72 

7.90 


Table  P-2.  Level  2  SA:  Means  and  Standard  Deviations  (percent). 


Question 


Next  Fix 
Flight  Profile 
Separation 
Open  Clearance 
Clearance  Received 
Clearance  Conformance 
In  Communication 
Flight  Plan  Conformance 


|  Low  Load 

|  High  Load 

Load  Coll. 

1  Low  Load  1 

High  Load  |  Load  Coll.  |  Low  Load  I  High  Load 


Load/Pos. 

Collapsed 


45.00  25 
38.54  71 
39.47  84 
30.18  16 
33.09  51 
34.16  48 
23.88  36 
29.21  99 


1  52.90  45.74 

39.88 

7  47.50  40.36 

60.07 

)  71.90  43.51 

78.39 

8  27.80  20.88 

22.20 

5  83.10  36.79 

68.90 

1  75.10  36.97 

62.64 

Next  Fix 

36.71  38.07 

31.25  34.67 

Flight  Profile 

73.50  39.88 

79.79  39.02 

Separation 

90.56  30.33 

76.30  35.26 

Open  Clearance 

38.33  31.34 

31.39  25.76 

Clearance  Received 

75.82  35.07 

87.09  30.62 

Clearance  Conformance 

73.69  34.98 

81.88  32.98 

In  Communication 

19.77  24.86 

15.41  16.13 

Flight  Plan  Conformance 

99.57  20.05 

99.71  19.94 

32.74  56. 
36.69  60. 
31.65  93. 
25.44  33. 
30.41  63. 
29.34  61. 
19.11  9. 
2.69  99. 


36.27  39. 
39.37  71. 
33.80  88. 
28.57  29. 
33.34  66. 
34.07  63. 
20.90  23. 
19.89  99. 


36.30  38.17 
99.20  24.82 


35.30  39.22 

80.50  45.51 
85.00  24.40 
39.00  31.87 
80.70  35.43 

78.90  35.55 

22.50  26.56 

99.90  2.45 


56.60  44.02 

60.40  32.83 

93.60  18.23 

33.40  32.34 

63.40  36.90 
61.30  36.53 

9.90  7.20 
99.10  2.73 


39.73  39.45 
71.24  41.21 
88.01  29.49 
29.77  27.71 
66.08  34.64 

63.73  34.25 
23.08  27.57 
99.57  14.42 


20.70  25.45 

77.90  35.90 

85.50  36.55 
24.00  19.28 

83.90  31.62 
82.00  31.95 
20.60  23.91 
99.80  2.80 

18.50  25.53 

60.40  32.83 

79.40  35.43 

36.50  32.29 
77.30  38.51 
71.10  36.62 

17.90  24.38 
97.20  2.82 


31.25  34.67 
62.85  37.21 
79.29  38.32 
29.56  24.44 


23.38  29.20 
73.71  41.23 
85.15  29.78 
35.21  28.13 

81.37  34.93 
76.54  35.79 

18.82  22.46 

99.83  12.54 
33.85  35.25 

68.82  35.01 
88.05  30.65 
32.18  28.64 
77.10  34.20 
75.58  33.47 

15.82  18.46 

99.37  2.71 


34.76  36.57 
72.12  39.46 
84.04  34.12 
32.28  27.32 

78.23  34.44 

74.24  34.50 

19.77  23.42 
99.41  18.61 


Level  3  Question 


Handoff  Needed 
Clearance  Needed 


Handoff  Needed 
Clearance  Needed 


Handoff  Needed 
Clearance  Needed 


Handoff  Needed 
Clearance  Needed 


Low  Load  |  High  Load 


Load/Pos. 

Collapsed 


20.21  23.11  8.52  13.43  14.39  19.43 
10.11  15.23  14.81  15.44  12.47  15.17 


8.78 

7.29 


17.10  16.43 

28.70  32.87 

15.26  17.86 

35.22  35.53 

16.80  21.31 

26.18  30.61 

12.86  15.39 

21.82  25.56 

19.00  26.86 

29.30  34.11 

10.80  10.43 

25.30  24.98 

14.95  20.20 

27.36  29.42 

13.93  21.24 

28.31  33.69 

10.74  9.95 

26.29  25.75 

17.30  24.96 

32.30  33.13 

20.00  25.54 

16.30  16.62 

18.68  24.86 

24.42  26.19 

19.05  24.44 

29.22  29.90 

20.74  27.39 

17.99  21.32 

16.58  23.49 

34.46  34.77 

16.02  18.36 

23.51  25.30 

16.30  20.97 
29.03  30.43 

16.59  22.22 

27.91  31.12 

14.79  18.99 
22.05  24.18 

Table  P-4.  Aircraft  Position:  ANOV A  Results. 


PHI 

EOS 

p-  level 

Position 

m 

15 

.007 

Load 

0.000 

0.016 

usmmm. 

1 

15 

.948 

Automation 

0.011 

0.011 

■at 

.368 

Position  X  Load 

0.003 

0.009 

0 

15 

.601 

Position  X  Automation 

0.016 

■Mil 

.291 

Load  X  Automation 

2.136 

2 

0 

.136 

Position  X  Load  X  Automation 

0.103 

2 

.903 

Table  P-5.  Callsign  Letter:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/7-level 

Position 

0.102 

m 

.489 

Load 

1 

m 

.035 

MEW 

2 

HI 

.171 

Position  X  Load 

0.051 

0.104 

0.495 

1 

15 

.492 

Position  X  Automation 

2 

.243 

Load  X  Automation 

■ng 

■KBS 

2 

HI 

.428 

Position  X  Load  X  Automation 

0.013 

0.144 

0.089 

2 

.915 

Table  P-6.  Callsign  Number:  ANOVA  Results. 


msm 

MS  Error 

F 

EMI 

02323 

Position 

HHEIE 

0.012 

1.029 

m 

m 

Load 

HHIE 

0.013 

0.971 

m 

m 

Automation 

0.014 

1.052 

2 

HI 

.362 

Position  X  Load 

HEBE 

15 

.334 

Position  X  Automation 

■HUE 

HKEiE 

30 

.377 

0.014 

H 

.368 

HI 

.373 

Table  P-7.  Altitude:  ANOVA  Results. 


IWI 

■■901 

mu 

IfpSl 

/7-level 

Position 

HEIM 

m 

15 

.491 

Load 

MHiME 

i^KIEEI 

15 

.723 

Automation 

30 

.708 

Position  X  Load 

■HUB 

15 

.510 

Position  X  Automation 

0.342 

.264 

Load  X  Automation 

HBEQ 

0.159 

.399 

Position  X  Load  X  Automation 

1 

0.170 

.711 

Table  P-8.  Speed:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

EMI 

Position 

■Hues! 

WKmME 

m 

15 

.158 

Load 

H 

15 

.922 

Automation 

■■ms 

H 

30 

.126 

Position  X  Load 

1 

15 

.991 

Position  X  Automation 

■ail 

.141 

Load  X  Automation 

Hi 

0 

1.000 

Position  X  Load  X  Automation 

0.000 

0.028 

0.004 

2 

HI 

.996 

P-3 


Load 


utomation 


Position  X  Load 


Position  X  Automation 


Load  X  Automation 
[Position  X  Load  X  Automation 


Table  P-1 1 .  Awareness  of  Aircraft  Turning:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/?-Ievel 

Position 

0.012 

0.300 

0.041 

1 

15 

.843 

Load 

0.104 

0.153 

HK2SS 

1 

15 

.422 

Automation 

0.002 

m 

—CUT 

Position  X  Load 

1 

15 

IHE 

Position  X  Automation 

0.498 

2 

n 

Load  X  Automation 

0.152 

0.176 

0.868 

m 

Mae 

■KME 

Position  X  Load  X  Automation 

0.094 

0.162 

0.579 

2 

m 

.566 

Table  P-12.  Aircraft  Type:  ANOVA  Results. 


Table  P-13.  Level  of  Control:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/?-level 

Position 

0.013 

0.012 

1.043 

1 

15 

.323 

Load 

1 

15 

.563 

Automation 

0.011 

0.012 

0.927 

Position  X  Load 

0.013 

0.016 

0.815 

1 

m 

0.013 

liliila 

KH] 

■KM1 

m 

m 

1 

Hfgg 

wm 

m 

1 

Table  P-14.  Next  Fix:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/?-level 

Position 

0.013 

0.479 

0.027 

1 

15 

.872 

Load 

mmm 

m 

m 

.445 

Automation 

■KIM.- 

■Bil 

.026 

Position  X  Load 

0.013 

0.195 

0.066 

1 

15 

.801 

Position  X  Automation 

■Km] 

■HEBE 

30 

.343 

Load  X  Automation 

HEUE 

30 

.071 

Position  X  Load  X  Automation 

0.321 

0.327 

0.982 

2 

30 

.386 

Table  P-15.  Flight  Profile:  ANOVA  Results. 


F 

df  1 

df  2 

/>-level 

Position 

wmm 

5.310 

1 

15 

.036 

Load 

msm 

0.007 

1 

15 

.935 

Automation 

0.104 

0.424 

hb 

2 

30 

.784 

Position  X  Load 

■KB 

1.735 

1 

15 

.208 

Position  X  Automation 

3.017 

2 

30 

.064 

Load  X  Automation 

0.688 

0.342 

mm 

.151 

Position  X  Load  X  Automation 

0.477 

0.407 

1.170 

2 

30 

.324 

Table  P-16.  Awareness  of  Aircraft  Separation:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df2 

p-level 

Position 

0.001 

“0.276 

0.003 

1 

15 

.954 

Load 

2.172 

0.425 

5.112 

1 

15 

.039 

Automation 

0.529 

0.221 

2.390 

2 

30 

.109 

Position  X  Load 

0.131 

0.189 

0.693 

1 

15 

.418 

Position  X  Automation 

0.165 

0.024 

2 

30 

.977 

Load  X  Automation 

■KBEg 

HEBE 

I^KBE] 

2 

30 

.550 

Position  X  Load  X  Automation 

2 

30 

.817 

Table  P-17.  Open  Clearances:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-level 

Position 

HESu] 

HKiSi 

1 

15 

.369 

Load 

15 

.448 

Automation 

HUB 

■■^g 

E 

30 

.589 

Position  X  Load 

0.062 

0.105 

0.590 

i 

15 

.454 

Position  X  Automation 

■■SEE 

mm! 

.456 

Load  X  Automation 

■K£H 

■ksie 

.910 

Position  X  Load  X  Automation  ’ 

■E 

P-5 


Table  P-18.  Clearances  Received:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-level 

Position 

0.700 

0.457 

1.533 

1 

15 

Load 

7.322 

1 

■yid 

Automation 

0.135 

0.315 

HKE££] 

■Hi! 

Position  X  Load 

0.215 

jg 

m 

■Kg 

Position  X  Automation 

0.225 

.287 

Load  X  Automation 

■K11EI 

HK1S& 

m 

Position  X  Load  X  Automation 

IHESSl 

m 

HE 

Table  P-19.  Clearance  Conformance:  ANOVA  Results. 


I  MS  Effect  I  MS  Error  I 


Position 


Load 


Automation  1 


Automation  2 


Automation  3 _ 

utomation 


Low  Load 


High  Load _ 

Position  x  Load 


Position  x  Automation 


Load  x  Automation _ 

osition  x  Load  x  Automation 


F |  df  1 1  df  2  j/7-leve!  | 


0.055 

0.114 

0.025 

0.214 

0.714 

0.253 

Table  P-20.  In  Communication:  ANOVA  Results. 


Table  P-21.  Flight  Plan  Conformance:  ANOVA  Results. 


MS  Effect 

. 

MS  Error 

F 

|[3I 

EE 

Position 

0.116 

0.145 

0.800 

■D 

m 

HHI 

Load 

0.050 

0.149 

0.333 

1 

ME 

■ 

Automation 

0.150 

2 

HE 

.234 

Position  X  Load 

0.116 

0.131 

i 

15 

.362 

Position  X  Automation 

0.139 

0.099 

1.407 

2 

30 

.261 

Load  X  Automation 

0.045 

0.087 

0.514 

2 

30 

.603 

Position  X  Load  X  Automation 

0.000 

0.116 

0.002 

2 

30 

.998 

P-6 


Table  P-22.  Level  3  SA:  MANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

F 

dfl 

df  2 

^-level 

Position 

.922 

0.591 

2 

14 

.567 

Load 

.918 

0.625 

2 

m 

.550 

.872 

■area 

mm 

msh 

me 

.777 

Position  X  Load 

■&r»r 

ME 

.500 

Position  X  Automation 

.673 

0.327 

KEi 

m 

m 

.275 

Load  X  Automation 

■ili EK 

m 

ME 

.960 

.663 

0.337 

0.509 

0.509 

1.526 

4 

12 

.256 

Table  P-23.  Handoff  Needed:  ANOVA  Results. 


F 

fitil 

■ 

Position 

■KMiE 

ME 

Load 

H 

ME 

Automation 

bkme 

m 

IE 

Position  X  Load 

WKKSM 

WMSBEft 

ME 

■E 

Position  X  Automation 

MMMi 

— rrrn 

Load  X  Automation 

WKKBM 

2 

Position  X  Load  X  Automation 

0.067 

0.113 

0.588 

2 

30 

.562 

Table  P-24.  Clearances  Needed:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

dfl 

df  2 

/?-level 

Position 

■BUIE 

imsEsk 

IHDEsu 

i 

ME 

WMffl 

Load 

HI 

■E 

Automation 

K 

■Qg} 

Position  X  Load 

hkme 

gg 

1 

ME 

Position  X  Automation 

0.274 

0.161 

■■mg 

2 

30 

Load  X  Automation 

0.039 

0.242 

0.160 

2 

30 

.853 

Position  X  Load  X  Automation 

0.021 

0.151 

0.140 

2 

30 

.870 

P-7 


APPENDIX  Q 
PSQ  SA  Results 


Table  Q-l.  PSQ  SA:  Means  and  Standard  Deviations. 


^'^sn-ai^SS 

Mg 

. 

Load/Pos. 

SA 

Question 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Collapsed 

■am 

Him 

j 

law 

Means] 

SDs 

Means  | 

SDs 

Means] 

SDs 

1 0)  Current  Aircraft  Locations 

7.37 

1.62 

6.62 

1.62 

K 

1.58 

6.50 

2.15 

6.12 

2.09 

6.31 

6.94 

1.93 

6.37 

1.85 

6.65 

1.90 

1 1)  Projected  Aircraft  Locations 

7.08 

1.75 

6.52 

1.66 

1.71 

6.95 

6.33 

2.24 

6.64 

2.16 

7.02 

1.91 

6.42 

1.94 

6.72 

1.93 

WBm 

12)  Potential  Violations 

7.48 

1.56 

6.48 

2.00 

6.98 

1.83 

6.79 

2.37 

6.29 

2.05 

6.54 

7.14 

2.00 

6.39 

1.99 

6.76 

2.02 

1 0)  Current  Aircraft  Locations 

7.09 

1.75 

6.62 

1.23 

6.86 

1.51 

6.47 

2.24 

5.75 

1.90 

6.11 

2.07 

6.78 

2.00 

6.18 

1.63 

6.48 

1.84 

1 1)  Projected  Aircraft  Locations 

7.43 

1.40 

6.72 

1.37 

7.07 

1.41 

6.43 

2.41 

6.22 

1.96 

6.32 

2.17 

6.93 

2.01 

6.47 

1.69 

6.70 

1.85 

12)  Potential  Violations 

7.53 

1.87 

6.90 

1.29 

7.21 

1.61 

6.78 

2.31 

6.71 

1.84 

6.75 

2.06 

7.16 

2.11 

6.80 

1.57 

6.98 

1.85 

10)  Current  Aircraft  Locations 

7.47 

1.42 

6.00 

2.02 

6.73 

1.88 

6.56 

2.09 

5.06 

2.04 

5.81 

2.01 

7.01 

1.82 

5.53 

2.05 

6.27 

2.06 

Auto  4 

1 1)  Projected  Aircraft  Locations 

7.68 

1.24 

5.89 

1.85 

6.78 

1.80 

7.14 

1.53 

5.45 

2.13 

6.30 

2.01 

7.41 

1.40 

5.67 

1.98 

6.54 

1.91 

■mi 

12)  Potential  Violations 

7.72 

1.36 

5.98 

2.06 

6.85 

1.93 

7.92 

1.67 

5.54 

2.30 

6.73 

2.31 

7.82 

1.50 

5.76 

2.16 

6.79 

2.11 

10)  Current  Aircraft  Locations 

7.31 

1.57 

6.41 

1.65 

6.86 

1.67 

6.51 

2.12 

5.64 

2.01 

6.08 

2.10 

6.91 

1.90 

6.03 

1.87 

6.47 

1.93 

MIT 

1 1)  Projected  Aircraft  Locations 

7.40 

1.47 

6.37 

1.65 

6.88 

1.64 

6.84 

2.03 

6.00 

2.10 

6.42 

2.10 

7.12 

1.78 

6.19 

1.89 

6.65 

1.89 

p|:% 

12)  Potential  Violations 

7.58 

1.58 

6.45 

1.81 

7.01 

1.78 

7.16 

2.16 

6.18 

2.08 

6.67 

2.17 

7.37 

1.90 

6.32 

1.95 

6.84 

1.99 

Table  Q-2.  Situational  Awareness:  MANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

F 

df  1 

E0 

Position 

.523 

0.477 

0.912 

0.912 

3.953 

3 

m 

Load 

■EE 

3 

■E 

IKES 

mtm 

iai 

HUH 

Position  X  Load 

.968 

0.032 

0.033 

■OOMtl 

gjJJjjc] 

m 

m 

Position  X  Automation 

.478 

0.522 

1.091 

QH 

m 

H 

I 

Load  X  Automation 

.475 

0.525 

1.106 

1.106 

1.844 

6 

Position  X  Load  X  Automation 

.691 

0.309 

0.447 

0.447 

0.745 

6 

Table  Q-3.  Situational  Awareness  for  Current  Aircraft:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df2 

/>-level 

Position 

huse 

1 

15 

mbhe 

Load 

eg 

ggpglg 

1 

■E 

HiEl 

BBEHI ■ 

2 

H 

Position  X  Load 

■E 

Position  X  Automation 

1 

m 

WBL 

Load  X  Automation 

m 

Position  X  Load  X  Automation 

0.403 

Table  Q-4.  Situational  Awareness  for  Projected  Aircraft:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p- level 

Position 

| KilEi 

WKmmm 

1 

ME 

Load 

■ 

HHESS3 

1 

ME 

Automation 

mmm 

2 

Position  X  Load 

2.537 

ML 

WMSM 

Position  X  Automation 

1.417 

■ 

■gi! 

.412 

Load  X  Automation 

7.888 

2.587 

2 

i: 

Position  X  Load  X  Automation 

0.335 

2.810 

2 

Q-l 


Table  Q-5.  Situational  Awareness  for  Potential  Violations:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/?-!evel 

Position 

5.616 

3.617 

1.553 

1 

15 

.232 

Load 

53.303 

2.400 

22.209 

1 

15 

.000 

Automation 

0.907 

2.495 

0.363 

2 

30 

.698 

Position  X  Load 

0.243 

3.470 

0.070 

1 

15 

.795 

Position  X  Automation 

0.596 

1.227 

0.486 

2 

30 

.620 

Load  X  Automation 

12.714 

3.087 

4.119 

2 

30 

.026 

Position  X  Load  X  Automation 

1.818 

3.625 

0.502 

2 

30 

.611 

Q-2 


APPENDIX  R 


SA  from  OTS 


Table  R-l .  SA  from  OTS:  Means  and  Standard  Deviations. 


Attention 

and  SA 

Question 

g  M—m 

SUBBiH 

Load/Pos. 

Collapsed 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Means  |  SDs 

Means)  SDs 

Means)  SDs 

Means  |  SDs 

Means  |  SDs 

Means)  SDs 

Means )  SDs 

Means  |  SDs 

Means  |  SDs 

gj 

4)  Awareness  of  Aircraft  Positions 

5)  Ensuring  Positive  Control 

6)  Pilot  Deviations  from  Control  Instructions 

7)  Correcting  Errors  in  a  Timely  Manner 

5.77  0.91 
6.38  0.96 
5.79  1.11 

6.07  1.06 

5.02  1.46 
5.38  1.54 
5.35  1.25 

5.44  1.37 

5.40  1.25 
5.88  1.36 
5.57  1.18 

5.75  1.25 

5.77  0.91 
6.38  0.96 
5.79  1.11 

6.07  1.06 

5.02  1.46 

5.38  1.54 

5.35  1.25 

5.44  1.37 

5.40  1.25 

5.88  1.36 

5.57  1.18 

5.75  1.25 

5.77  0.89 

6.38  0.94 
5.79  1.09 

6.07  1.05 

5.02  1.43 

5.38  1.52 
5.35  1.23 
5.44  1.34 

5.40  1.24 

5.88  1.35 

5.57  1.17 

5.75  1.24 

Is 

4)  Awareness  of  Aircraft  Positions 

5)  Ensuring  Positive  Control 

6)  Pilot  Deviations  from  Control  Instructions 

7)  Correcting  Errors  in  a  Timely  Manner 

6.19  0.62 
6.19  0.65 

5.97  0.37 

6.13  0.62 

5.36  1.18 
6.00  0.63 
5.71  0.99 

5.94  0.77 

5.77  1.02 
6.10  0.64 
5.84  0.75 

6.04  0.69 

6.19  0.62 
6.19  0.65 
5.97  0.37 

6.13  0.62 

5.36  1.18 
6.00  0.63 

5.71  0.99 

5.94  0.77 

5.77  1.02 
6.10  0.64 
5.84  0.75 

6.04  0.69 

6.19  0.61 
6.19  0.64 
5.97  0.36 

6.13  0.61 

5.36  1.16 
6.00  0.62 
5.71  0.98 

5.94  0.76 

5.77  1.01 
6.10  0.64 
5.84  0.74 

6.04  0.69 

n 

4)  Awareness  of  Aircraft  Positions 

5)  Ensuring  Positive  Control 

6)  Pilot  Deviations  from  Control  Instructions 

7)  Correcting  Errors  in  a  Timely  Manner 

6.50  0.87 
6.44  0.51 
6.28  0.62 

6.69  0.47 

5.21  1.16 

5.69  0.95 
5.92  1.44 

5.88  0.96 

5.86  1.20 
6.06  0.84 
6.10  1.10 
6.29  0.85 

6.50  0.87 

6.44  0.51 
6.28  0.62 

6.69  0.47 

5.21  1.16 

5.69  0.95 
5.92  1.44 

5.88  0.96 

5.86  1.20 
6.06  0.84 
6.10  1.10 

6.29  0.85 

6.50  0.85 
6.44  0.50 
6.28  0.61 

6.69  0.46 

5.21  1.14 

5.69  0.93 
5.92  1.41 

5.88  0.94 

5.86  1.19 

6.06  0.83 
6.10  1.10 

6.29  0.84 

P 

4)  Awareness  of  Aircraft  Positions 

5)  Ensuring  Positive  Control 

6)  Pilot  Deviations  from  Control  Instructions 

7)  Correcting  Errors  in  a  Timely  Manner 

6.15  0.85 

6.34  0.72 

6.01  0.77 

6.30  0.80 

5.20  1.25 
5.69  1.11 
5.66  1.23 

5.75  1.06 

5.68  1.17 
6.01  0.99 
5.84  1.04 

6.03  0.97 

6.15  0.85 

6.34  0.72 

6.01  0.77 

6.30  0.80 

5.20  1.25 

5.69  1.11 
5.66  1.23 

5.75  1.06 

5.68  1.17 

6.01  0.99 
5.84  1.04 

6.03  0.97 

6.15  0.84 

1  6.34  0.72 
|  6.01  0.77 
6.30  0.79 

5.20  1.25 
5.69  1.11 

5.66  1.23 

5.75  1.06 

5.68  1.16 
6.01  0.99 
5.84  1.04 

6.03  0.97 

Table  R-2.  Maintaining  Attention  and  SA:  MANOVA  Results. 


Wilks  A 

Pillai’s 

Hotelling 

Roy’s 

F 

df  1 

df  2 

jp-level 

Load 

.074 

0.926 

12.517 

12.517 

12.517 

4 

4 

.016 

Table  R-3.  Maintaining  Awareness  of  Aircraft  Positions:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

/7-level 

Load 

11.021 

0.271 

4.692 

1 

7 

.000 

Automation 

0.956 

0.569 

1.680 

2 

14 

.222 

Load  X  Automation 

0.341 

1.090 

0.313 

2 

14 

.736 

Table  R-4.  Ensuring  Positive  Control:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

/7-level 

Load 

5.005 

.089 

56.529 

1 

7 

.000 

Automation 

0.226 

0.772 

0.292 

2 

14; 

.751 

Load  X  Automation 

0.694 

0.562 

1.236 

2 

14 

.320 

Table  R-5.  Detecting  Pilot  Deviations  From  Control  Instructions:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

/7-level 

Load 

1.505 

0.350 

4.295 

1 

7 

.077 

Automation 

1.107 

0.539 

2.054 

2 

14 

.165 

Load  X  Automation 

0.032 

0.459 

0.069 

2 

14 

.933 

Table  R-6.  Correcting  Own  Errors  in  a  Timely  Manner:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

3.521 

0.176 

2.051 

1 

7 

.003 

Automation 

1.134 

0.459 

.  2.473 

2 

14 

.120 

Load  X  Automation 

0.415 

0.371 

1.119 

2 

14 

.354 

R-l 


APPENDIX  S 


ATWIT  Results 


Table  S-l.  ATWIT  Ratings:  Means  and  Standard  Deviations. 


Ratings 


Interval 

Low  Load 

Means  |  SDs 

1 

3.01  1.64 

2 

3.27  1.70 

3 

3.46  1.94 

4 

3.60  1.91 

5 

3.59  1.76 

6 

3.21  1.65 

7 

3.13  1.86 

8 

3.24  2.14 

9 

3.19  1.90 

10 

3.04  1.83 

Interval  Collapsed 

3.27  1.80 

1 

2.95  1.20 

2 

2.66  1.35 

3 

2.78  1.51 

4 

3.06  1.37 

5 

2.73  1.35 

6 

2.84  1.36 

7 

2.44  1.37 

8 

2.30  1.53 

9 

2.19  1.38 

10 

2.16  1.43 

Interval  Collapsed 

2.61  1.38 

1 

3.74  1.98 

2 

3.49  2.00 

3 

3.29  1.35 

4 

3.71  1.60 

5 

3.63  1.73 

6 

3.55  2.14 

7 

3.54  2.20 

8 

2.99  1.68 

9 

3.13  2.33 

10 

2.65  2.11 

Interval  Collapsed 

3.42  1.85 

1 

3.23  1.63 

2 

3.13  1.70 

3 

3.17  1.62 

4 

3.45  1.63 

5 

3.31  1.64 

6 

3.19  1.72 

7 

3.02  1.85 

8 

2.84  1.81 

9 

2.83  1.91 

10 

2.62  1.81 

Interval  Collapsed 

3.10  1.72 

High  Load 


4.82  2.00 
5.02  1.96 
5.16  2.01 
5.23  1.95 
5.46  2.32 
5.27  1.47 
5.50  2.36 
5.18  2.29 
5.25  2.24 
5.67  2.20 


5.26  2.05 


5.62  1.75 
5.78  1.86 
5.50  1.97 
5.73  2.18 
5.71  1.99 

5.56  1.57 
5.80  1.61 

5.66  1.51 

5.67  1.72 
5.37  1.39 


5.54  1.71 


5.37  2.22 
5.50  2.25 
5.79  2.20 
5.76  2.13 

5.63  2.29 
5.97  2.46 
5.71  2.47 

5.64  2.48 
6.00  2.48 
5.91  2.53 


5.83  2.33 


5.26  2.00 
5.43  2.02 
5.49  2.04 
5.57  2.06 

5.60  2.17 

5.61  1.89 
5.67  2.15 
5.49  2.12 
5.64  2.16 
5.66  2.09 


5.54  2.05 


Load  Coll. 

Low  Load 

High  Load 

Means  |  SDs 

Means  |  SDs 

Means  |  SDs 

3.92 

2.02 

2.64 

1.22 

4.70 

1.62 

4.15 

2.01 

2.52 

1.29 

4.65 

1.66 

4.31 

2.13 

2.65 

1.43 

4.41 

1.20 

4.42 

2.07 

2.79 

1.23 

4.44 

1.44 

.4.53 

2.24 

2.84 

1.36 

4.29 

1.39 

4.24 

1.86 

2.72 

1.43 

4.78 

1.42 

4.32 

2.42 

2.57 

1.51 

4.44 

1.63 

4.21 

2.39 

2.49 

1.45 

4.12 

1.41 

4.22 

2.30 

2.69 

1.54 

4.31 

1.74 

4.36 

2.40 

2.48 

1.62 

4.22 

1.53 

4.27 

.2.17 

2.64 

1.38 

4.43 

1.48 

4.24 

1.99 

2.83 

1.29 

4.75 

1.27 

4.17 

2.25 

2.59 

1.45 

5.50 

1.01 

4.10 

2.21 

2.78 

1.56 

5.03 

1.49 

4.35 

2.23 

3.19 

1.89 

5.13 

1.49 

4.17 

2.25 

2.73 

1.25 

5.11 

1.35 

4.16 

1.99 

2.72 

1.48 

5.10 

1.41 

4.07 

2.25 

2.63 

1.41 

5.07 

1.38 

3.93 

2.26 

2.30 

1.44 

5.19 

1.27 

‘3.87 

2.33 

2.37 

1.50 

5.33 

1.68 

3.71 

2.14 

2.35 

1.37 

5.44 

1.83 

4.08 

2.13 

2.65 

1.45 

5.10 

1.39 

4.61 

2.24 

3.48 

2.07 

4.96 

1.59 

4.56 

2.34 

3.09 

1.30 

5.23 

1.29 

4.62 

2.22 

2.89 

1.21 

5.38 

1.47 

4.80 

1.49 

3.18 

1.49 

5.41 

1.49 

4.69 

2.26 

3.16 

1.62 

5.57 

1.81 

4.84 

2.59 

2.69 

1.65 

5.33 

2.00 

4.69 

2.56 

2.74 

1.58 

4.89 

1.69 

4.40 

2.50 

2.93 

1.66 

5.05 

1.72 

4.66 

2.78 

2.53 

1.64 

5.47 

2.21 

4.38 

2.84 

2.45 

1.67 

5.20 

1.78 

4.62 

2.43 

2.99 

1.56 

5.32 

1.72 

4.32 

2.04 

2.97 

1.57 

4.81 

1.48 

4.39 

2.14 

2.73 

1.34 

5.12 

1.37 

4.44 

2.14 

2.77 

1.38 

4.95 

1.42 

4.62 

2.09 

3.05 

1.54 

5.00 

1.50 

4.50 

2.20 

2.90 

1.39 

5.00 

1.60 

4.45 

2.14 

2.71 

1.49 

5.07 

1.63 

4.38 

2.34 

2.64 

1.47 

4.80 

1.57 

4.23 

2.33 

2.57 

1.51 

4.78 

1.53 

4,24 

2.42 

2.53 

1.53 

5.04 

1.94 

4.11 

2.42 

2.42 

1.52 

4.95 

1.76 

4.32 

2.26 

2.76 

1.47 

4.95 

1.58 

1.75  2.82 
1.82  2.90 

1.58  3.05 
1.56  3.20 
1.54  3.21 
1.75  2.96 
1.81  2.85 
1.63  2.87 
1.81  2.94 
1.78  .  2.76 
1.69  2.96 

1.59  2.89 
1.92  2.63 


1.43  4.76 
1.54  4.83 
1.73  4.78 
1.63  4.83 
1.59  4.88 
1.54  5.02 
1.69  4.97 
1.84  4.65 
1.72  4.78 
1.72  4.95 


1.63  4.85 
.23  5.18 
.38  5.64 
.51  5.27 


1.88  2.63 


1.95  3.61 
1.67  3.29 

1.84  3.09 

1.85  3.44 
2.09  3.40 
2.26  3.12 
1.95  3.14 
1.98  2.96 
2.44  2.83 
2.20  2.55 


2.01  3.20 

1.78  3.10 

1.79  2.93 

1.75  2.97 

1.76  3.25 

1.80  3.11 
1.93  2.95 
1.86  2.83 
1.84  2.71 
2.11  2.68 
2.08  2.52 


1.88  2.93 


1.42 


2.00  5.16 
1.67  5.36 
1.28  5.59 
1.54  5.59 
1.66  5.60 
1.93  5.65 
1.93  5.30 
1.65  5.34 
2.00  5.73 
1.87  5.56 


1.72  5.58 
1.60  5.04 
1.54  5.27 
1.51  5.22 
1.59  5.29 
1.53  5.30 
1.62  5.34 
1.67  5.23 
1.66  5.14 

1.73  5.34 
1.66  5.31 


1.61  5.25 


Load/Pos. 


1.79  3.79  1.88 

1.80  3.86  1.92 
1.67  3.92  1.90 
1.74  4.01  1.86 
1.97  4.05  1.97 
1.44  3.99  1.81 
2.07  3.91  2.16 
1.95  3.76  2.08 
2.03  3.86  2.08 
2.00  3.85  2.16 


1.83  3.90  1.97 
1.56  4.00  1.81 
1.48  4.08  2.08 
1.73  3.98  2.04 
4.24  2.08 
4.03  2.01 
4.01  1.92 
3.94  2.05 
3.81  2.12 
3.84  2.23 
3.78  2.17 


1.91  4.44  2.09 

1.81  4.39  2.02 
1.85  4.42  2.03 

1.82  4.58  2.00 
2.03  4.57  2.16 
2.23  4.46  2.44 
2.12  4.29  2.29 
2.12  4.23  2.25 
2.33  4.37  2.61 
2.19  4.15  2.53 


2.06  4.39  2.24 


1.76  4.08  1.94 
1.72  4.11  2.01 

1.77  4.11  1.99 
1.82  4.28  1.99 
1.92  4.21  2.05 

1.78  4.16  2.08 
1.92  4.05  2.16 
1.87  3.93  2.15 
2.06  4.03  2.32 
1.95  3.93  2.29 


4.09  2.09 


Table  S-2.  ATWIT  Response  Times:  Means  and  Standard  Deviations  (seconds). 


R  Times 

Low  Load 

Pl-IighLoad 

Load^  Coll. 

Low  Load 

ES5 

Higl 

Load 

|  Load  Col!. 

f  Low  Load 

Collapsed 

Load/Pos. 

Collapsed 

EgJTJgj 

■ana 

Bl 

Rfwl 

IISBM 

B33H 

Meansl 

SDs 

1 

1.92 

0.74 

2.19 

1.00 

2.06 

0.88 

2.32 

0.94 

2.75 

1.19 

2.53 

1.08 

2.12 

0.86 

2.47 

1.12 

2.30 

1.00 

2 

1.75 

1.06 

1.65 

1.30 

1.70 

1.17 

2.35 

0.70 

2.63 

0.87 

2.49 

0.79 

2.05 

0.94 

2.14 

1.20 

2.10 

1.07 

3 

2.38 

1.11 

1.93 

1.05 

2.15 

1.09 

2.21 

0.71 

2.57 

0.84 

2.39 

0.79 

2.30 

0.92 

2.25 

0.99 

2.27 

0.95 

«*■* 

4 

1.91 

1.17 

2.27 

1.10 

2.08 

1.13 

2.21 

0.78 

2.53 

1.44 

2.37 

1.15 

2.05 

0.99 

2.40 

1.27 

2.22 

1.14 

6 

5 

2.08 

1.32 

1.84 

1.00 

1.96 

1.16 

2.88 

1.96 

2.69 

1.02 

2.78 

1.54 

2.48 

1.69 

2.27 

1.08 

2.37 

1.41 

n 

C 

6 

1.88 

0.77 

2.57 

1.64 

2.22 

1.86 

2.21 

0.72 

3.20 

3.35 

2.70 

2.43 

2.04 

0.75 

2.88 

2.61 

2.46 

1.95 

3-  .  ' 

7 

2.06 

1.11 

2.46 

2.85 

2.26 

2.13 

3.48 

4.53 

2.40 

0.77 

2.94 

3.24 

2.77 

3.32 

2.43 

2.05 

2.60 

2.74 

'f 

y. 

8 

2.07 

0.75 

2.35 

1.52 

2.21 

1.19 

2.65 

2.60 

2.20 

0.84 

2.42 

1.92 

2.36 

1.91 

2.27 

1.21 

2.32 

1.58 

9 

2.30 

1.44 

2.37 

1.38 

2.33 

1.39 

3.87 

3.91 

2.54 

1.27 

3.21 

2.94 

3.08 

3.01 

2.46 

1.31 

1.59 

0.68 

2.47 

0.94 

2.03 

0.92 

2.61 

1.18 

2.42 

1.13 

2.52 

1.14 

2.10 

1.08 

2.45 

1.02 

Wm 

Interval  Collapsed 

1.99 

1.04 

2.21 

1.47 

2.10 

1.27 

2.68 

2.26 

2.59 

1.44 

2.64 

1.89 

2.34 

1.79 

2.40 

1.46 

2.37 

1.63 

1 

1.66 

0.80 

2.32 

1.18 

1.98 

1.04 

2.93 

1.54 

2.70 

0.67 

2.82 

1.19 

2.30 

1.37 

2.51 

0.96 

2.40 

118 

2 

2.02 

0.64 

2.34 

1.68 

2.17 

1.24 

2.28 

0.57 

2.49 

1.04 

2.38 

0.83 

2.15 

0.61 

2.41 

1.37 

2.28 

1.05 

3 

2.08 

1.92 

2.05 

0.48 

2.07 

1.40 

2.47 

1.25 

2.36 

1.06 

2.42 

1.15 

2.28 

1.61 

2.20 

0.82 

2.24 

1.28 

M 

Pj  ... 

4 

1.64 

0.89 

1.86 

0.83 

1.75 

0.85 

2.34 

1.24 

2.03 

0.65 

2.19 

0.99 

1.99 

1.12 

1.94 

0.74 

1.97 

0.94 

5 

1.62 

0.72 

2.10 

0.91 

1.85 

0.84 

2.34 

1.30 

2.49 

1.14 

2.41 

1.21 

1.98 

1.10 

2.29 

1.03 

2.13 

1.07 

5'  : 

6 

2.69 

1.33 

2.39 

1.05 

2.54 

1.19 

2.79 

1.38 

2.85 

1.40 

2.82 

1.37 

2.74 

1.33 

2.62 

1.24 

2.68 

1.28 

£ _ _ 

7 

2.07 

0.92 

2.25 

0.97 

2.16 

0.93 

2.57 

1.37 

2.31 

0.84 

2.44 

1.14 

2.32 

1.18 

2.28 

0.90 

2.30 

1.04 

< 

8 

1.77 

1.04 

2.49 

1.12 

2.12 

1.12 

2.51 

1.38 

2.24 

0.59 

2.38 

1.07 

2.14 

1.26 

2.37 

0.89 

2.25 

1.09 

1 

9 

2.98 

4.02 

2.12 

1.03 

2.57 

2.96 

2.37 

0.71 

3.00 

1.86 

2.67 

1.40 

2.68 

2.86 

2.56 

1.54 

2.62 

2.30 

~  V‘t 

10 

1.90 

0.93 

2.62 

1.15 

2.24 

1.09 

2.74 

1.80 

2.81 

1.42 

2.77 

1.60 

2.32 

1.47 

2.71 

1.27 

2.51 

1.38 

iiL;-:. 

Interval  Collapsed 

2.04 

1.65 

2.27 

1.03 

2.15 

1.38 

2.5.3 

1.28 

2.52 

1.10 

2.53 

1.19 

2.29 

1.49 

2.39 

1.08 

2.34 

1.30 

:a;1| 

1 

1.91 

0.77 

2.14 

2.03 

mm 

3.65 

2.77 

2.87 

1.14 

3.23 

2.07 

2.78 

2.18 

2.50 

1.01 

2.63 

1.66 

•"  :  ’  •. '  tra 

2 

1.96 

lira 

2.94 

2.35 

2.27 

4.09 

3.46 

2.44 

0.75 

3.22 

2.53 

3.02 

2.74 

2.57 

2.12 

2.78 

2.42 

3 

2.20 

1.14 

1.64 

1.90 

1 MU 

3.13 

1.81 

3.43 

3.35 

3.29 

2.70 

2.67 

1.56 

2.53 

2.53 

2.60 

2.12 

. . » ■:  i 

4 

wm 

1.95 

1.49 

3.70 

1.57 

2.47 

1.12 

3.04 

1.47 

2.88 

1.93 

2.28 

1.05 

2.56 

1.54 

jsy 

5 

1.31 

2.44 

1.49 

2.25 

2.68 

0.67 

2.14 

0.61 

2.39 

0.69 

2.36 

1.08 

2.29 

1.13 

2.32 

1.10 

a  ':i' 

5  . 

6 

2.44 

1.37 

2.28 

1.16 

2.36 

1.25 

2.73 

1.15 

2.27 

0.58 

2.49 

0.91 

2.59 

1.25 

2.28 

0.90 

2.42 

1.08 

©  -  • 

a 

7 

2.48 

1.52 

2.29 

1.47 

2.38 

1.47 

2.80 

0.94 

2.39 

0.77 

2.58 

0.86 

2.64 

1.25 

2.34 

1.16 

2.48 

1.20 

<  ' 

8 

2.36 

feJB 

2.11 

1.55 

3.30 

1.75 

3.05 

1.56 

3.17 

1.63 

2.83 

1.93 

2.47 

1.40 

2.64 

1.67 

..  : 

9 

2.55 

1.47 

2.89 

1.56 

2.73 

1.51 

2.96 

1.15 

2.27 

0.74 

2.59 

1.00 

2.76 

1.31 

2.58 

1.24 

2.66 

1.27 

•  v 

1.52 

vm 

3.46 

4.43 

2.55 

3.37 

2.77 

0.73 

2.78 

1.42 

2.77 

1.13 

2.14 

0.95 

3.12 

3.26 

2.66 

2.49 

bniu ;  jiiuSuid 

Interval  Collapsed 

2.17 

1.35 

2.38 

2.05 

2.28 

1.74 

3.13 

1.77 

2.62 

1.49 

2.88 

1.66 

2.65 

1.64 

2.50 

1.80 

2.58 

1.72 

PT 

'  l!r  -•  ’i| 

1 

1.83 

0.76 

2.21 

0.97 

2.04 

0.91 

2.95 

1.92 

2.77 

1.01 

2.82 

1.49 

2.39 

1.56 

2.49 

1.02 

2.44 

1.31 

pi 

■ 

*•<  ■;  ci-viiiii 

2 

1.91 

0.94 

2.23 

2.13 

2.16 

1.66 

2.88 

2.14 

2.52 

0.87 

2.71 

1.63 

2.40 

1.72 

2.38 

1.62 

2.39 

1.67 

u 

p;i; 

W  .  .  : 

yo-  ji® 

3 

2.22 

1.42 

1.86 

0.74 

2.06 

1.17 

2.60 

1.35 

2.81 

2.15 

2.73 

1.77 

2.41 

1.39 

2.34 

1.67 

2.37 

1.53 

«;•**- 

4 

1.87 

1.38 

2.08 

0.97 

1.97 

1.18 

2.73 

1.38 

2.35 

1.12 

2.50 

1.26 

2.30 

1.44 

2.21 

1.05 

2.26 

1.26 

.  c  ...-H 

5 

1.91 

1.14 

2.13 

1.18 

1.98 

1.14 

2.63 

1.41 

2.43 

0.95 

2.50 

1.20 

2.27 

1.33 

2.28 

1.08 

2.28 

1.20 

c  . ; ;  [f  | 

6 

2.33 

1.21 

2.41 

1.29 

2.35 

1.24 

2.57 

1.12 

2.76 

2.10 

2.62 

1.65 

2.45 

1.17 

2.59 

1.74 

2.52 

1.48 

.  a  ’ 

s 

7 

2.20 

1.19 

2.33 

1.90 

2.27 

1.60 

2.95 

2.78 

2.37 

0.78 

2.63 

1.99 

2.58 

2.16 

2.35 

1.44 

2.46 

1.83 

S  "  - 

8 

2.06 

1.37 

2.23 

1.21 

2.14 

1.27 

2.81 

1.97 

2.51 

1.15 

2.62 

1.58 

2.44 

1.73 

2.37 

1.18 

2.40 

1.48 

:'.Ai 

9 

2.61 

2.59 

2.48 

1.36 

2.49 

1.98 

3.07 

2.44 

2.59 

1.35 

2.85 

1.95 

2.84 

2.51 

2.53 

1.35 

2.68 

2.01 

1.67 

0.79 

2.87 

2.75 

2.26 

2.03 

2.70 

1.30 

2.67 

1.31 

2.70 

1.30 

2.19 

1.19 

2.77 

2.15 

2.48 

1.76 

.a vlarcis 

Interval  Collapsed  | 

2.07 

1.37 

2.28 

1.57 

2.18 

1.48 

2.78 

1.83 

2.58 

1.36 

2.68 

1.61 

2.43 

1.65 

2.43 

1.47 

EE2l 

S-2 


Table  S-3.  Mean  ATWIT  Ratings  and  Response  Times:  MANOVA  Results. 


I  Wilks  A I  Pillai's  I  Hotelling  I  Roy's 


df  1 1  df  2  Ip-level 


.001 

.000 


Table  S-7.  Mean  ATWIT  Ratings:  ANOVA  Results  (N=6). 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-level 

Position 

13.258 

4.037 

3.284 

1 

5 

.130 

Low  Load 

0.403 

0.647 

0.623 

1 

5 

.466 

High  Load 

5.468 

0.792 

6.905 

1 

5 

.047 

Load 

10.750 

1.436 

7.163 

1 

5 

.000 

R  -  Side 

24.499 

0.232 

105.708 

1 

5 

.000 

D  -  Side 

1.536 

0.340 

31.000 

1 

5 

.003 

Automation 

1.221 

0.924 

1.321 

2 

10 

.310 

Position  x  Load 

4.354 

0.279 

15.610 

1 

5 

.011 

Position  x  Automation 

0.748 

1.462 

0.512 

2 

10 

.614 

Load  x  Automation 

0.002 

0.700 

0.003 

2 

10 

.997 

Position  x  Load  x  Automation 

0.106 

1.071 

0.099 

2 

10 

.907 

Table  S-8.  Mean  ATWIT  Response  Times:  ANOVA  Results  (N=6). 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

<L> 

> 

JV 

1 

Cl, 

Position 

3.466 

0.680 

5.096 

1 

5 

.074 

Load 

0.301 

0.838 

0.360 

1 

5 

.575 

Automation 

0.632 

0.281 

2.250 

2 

10 

.156 

Position  X  Load 

1.304 

0.387 

3.372 

1 

5 

.126 

Position  X  Automation 

0.159 

0.157 

1.010 

2 

10 

.399 

Load  X  Automation 

0.411 

0.315 

1.304 

2 

10 

.314 

Position  X  Load  X  Automation 

0.021 

0.390 

0.054 

2 

10 

.948 

Table  S-9.  Maximum  ATWIT  Ratings:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/?-level 

Position 

8.689 

3.201 

2.715 

1 

15 

.120 

Load 

317.354 

2.186 

145.178 

1 

15 

.000 

Automation 

6.993 

2.032 

3.441 

2 

30 

.045 

Position  X  Load 

0.901 

2.028 

0.445 

1 

15l 

.515 

Position  X  Automation 

1.757 

1.409 

1.247 

2 

30 

.302 

Load  X  Automation 

2.609 

2.022 

1.290 

2 

30 

.290 

Position  X  Load  X  Automation 

0.483 

3.773 

0.128 

2 

3q 

.880 

Table  S-10.  Maximum  ATWIT  Ratings:  ANOVA  Results  (N=6). 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-level 

Position 

16.056 

2.922 

5.494 

1 

5 

.066 

Load 

117.556 

1.556 

75.571 

1 

5 

.000 

Automation 

1.542 

0.958 

1.609 

2 

10 

.248 

Position  X  Load 

5.556 

2.022 

2.747 

1 

— — i 

5 

.158 

Position  X  Automation 

1.431 

1.747 

6.819 

2 

id 

.468 

Load  X  Automation 

0.514 

0.964 

0.533 

2 

id 

.603 

Position  X  Load  X  Automation 

0.014 

3.631 

0.004 

2 

10 1 

.996 

S-4 


Table  S-ll.  Mean  ATWIT  Ratings  by  Time:  ANOVA  Results. 


Position 


Interval 


MS  Effect  MS  Error 
104.605  26.742" 


|  2.567|  1.414] 


F  dfl  df2  /7-level 
3.912  f  15  .06 


1.815]  9|  135]  .071 


Interval  1 

29.416 

Interval  2 

43.008 

Interval  3 

41.443 

Interval  4 

Interval  5 _ 

Interval  6 


Interval  7 
Interval  8 
Interval  9 
Interval  10 


-utomation _ 

Position  x  Interval 


Interval  x  Load _ 

Position  x  Automation 


nterval  x  Automation 


Load  x  Automation _ 

Position  x  Interval  x  Load _ 

}osition  x  Interval  x  Automation 


Position  x  Load  x  Automation _ 

Interval  x  Load  x  Automation _ 

Position  x  Interval  x  Load  x  Automation 


47.119 

45.519 

56.478 

61.636 


1  15  .000 


1  15  .000 


13.941  1  15  .000 


0.402]  117.230]  1]  15]  .000] 


Table  S-12.  Mean  ATWIT  Response  Times  by  Time:  ANOVA  Results. 


IMS  Effect! 

MS  Error 

i  F 

Position 

125.673 

Low  Load 

4.572 

High  Load 

0.547 

Interval 

3.088 

A 

Load 

0.391 

i 

R  -  Side 


Position  x  Load 

Interval  x  Load _ 

Position  x  Automation 


Load  x  Automation 

4.107 

L 

Position  x  Interval  x  Load 

2.228 

Position  x  Interval  x  Automation 

2.185 

Position  x  Interval  x  Load  x  Automation 


Table  S-13.  Mean  ATWIT  Ratings  by  Time:  ANO V A  Results  (N=6). 


MS  Effect 

MS  Error 

F 

dfl 

df  2 

/?-leveI 

Position 

132.582 

4.371 

3.284 

1 

5 

.130 

Low  Load 

0.623 

1 

5 

High  Load 

■BSEE 

m 

m 

B5SI 

Interval 

K 

m 

liHa 

Load 

1007.501 

14.359 

m 

m 

R  -  Side 

■■IEEE 

0.232 

105.708 

m 

m 

■E2E 

D  -  Side 

■HUE 

0.340 

■bee&e 

i 

5 

■ 

Automation 

12.206 

9.243 

2 

m 

Position  x  Interval 

^BiBE 

1.110 

9 

m 

— E35E1 

Position  x  Load 

■ 

2.789 

BPHE 

1 

5 

■ 

Interval  x  Load 

0.954 

9 

45 

Position  x  Automation 

BBS jg 

BEE££ 

m 

m 

.614 

Interval  x  Automation 

m 

be 

.914 

Load  x  Automation 

m 

HESS 

Position  x  Interval  x  Load 

wmm 

m 

m 

jg 

Position  x  Interval  x  Automation 

bm 

m 

be 

Position  x  Load  x  Automation 

■uifi 

m 

m 

Bi 

Interval  x  Load  x  Automation 

1.396 

.914 

1.527 

18 

be 

Position  x  Interval  x  Load  x  Automation 

0.411 

0.946 

0.434 

18 

90 

.976 

Table  S-14.  Mean  ATWIT  Response  Times  by  Time:  ANOVA  Results  (N=6). 


MS  Effect 

MS  Error 

F 

dfl 

p-Ievel 

Position 

34.665 

6.803 

5.096 

■E 

5 

■ 

Interval 

9 

45 

.174 

Load 

3.013 

8.379 

0.360 

1 

5 

.575 

Automation 

6.323 

2.811 

2.250 

2 

10 

.156 

Position  X  Interval 

2.363 

2.540 

0.931 

9 

45 

.508 

Position  X  Load 

13.036 

3.866 

3.372 

1 

5 

.126 

Interval  X  Load 

0.634 

2.254 

0.281 

9 

45 

.977 

Position  x  Automation 

1.587 

1.571 

1.010 

2 

10 

.399 

Interval  x  Automation 

1.371 

1.849 

0.741 

18 

90 

.760 

Load  x  Automation 

4.106 

3.149 

1.304 

2 

10 

.314 

Position  x  Interval  x  Load 

1.917 

2.484 

0.772 

9 

45 

.643 

Position  x  Interval  x  Automation 

1.621 

1.913 

0.847 

18 

90 

.641 

Position  x  Load  x  Automation 

0.209 

3.905 

0.054 

2 

10 

.948 

Interval  x  Load  x  Automation 

2.064 

1.676 

1.231 

18 

90 

.255 

Position  x  Interval  x  Load  x  Automation 

2.252 

2.455 

0.917 

18 

90 

.560 
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APPENDIX  T 
NASA  TLX  Results 

Table  T-l.  NASA  TLX:  Means  and  Standard  Deviations. 


NASA 

TLX 


Low  Load  I  W^iLOTdT  Load  Coll  Low  Load  |  High  Load  Load  Coll.  Low  Load  |  High  Load  Collapsed 


1)  Mental  Demand 

4.23 

1.55 

2)  Physical  Demand 

2.48 

1.39 

3)  Temporal  Demand 

4.08 

1.74 

4)  Performance 

7.75 

1.00 

5)  Effort 

6.20 

2.07 

6)  Frustration 

2.20 

1.61 

1)  Mental  Demand 

3.44 

1.78 

2)  Physical  Demand 

2.06 

0.90 

3)  Temporal  Demand 

3.33 

1.60 

4)  Performance 

7.94 

1.34 

5)  Effort 

5.67 

2.29 

6)  Frustration 

1.84 

0.94 

1)  Mental  Demand 

4.32 

2.32 

2)  Physical  Demand 

2.56 

1.99 

3)  Temporal  Demand 

4.21 

2.05 

4)  Performance 

7.88 

1.09 

5)  Effort 

5.61 

2.49 

6)  Frustration 

1  2.02 

1.23 

1)  Mental  Demand 

4.00 

1.91 

2)  Physical  Demand 

2.37 

1.48 

3)  Temporal  Demand 

3.87 

1.81 

4)  Performance 

7.86 

1.13 

5)  Effort 

5.83 

2.25 

6)  Frustration 

2.02 

1.27 

6.78  1.77  5.11  2.43  3.38  1.78  6.72  1.40 

4.08  2.44  3.07  2.08  2.38  1.57  3.77  1.93 

6.15  2.05  4.74  2.31  3.21  1.41  6.34  1.68 

7.00  1.51  7.47  1.48  6.44  2.28  7.19  1.38 

7.34  1.50  6.50  2.08  5.36  2.37  7.27  1.35 

3.66  1.70  2.75  1.64  2.52  1.68  4.34  2.31 


4.98 

2.17 

3.98 

1.75 

6.45 

1.81 

5.22 

2.16 

3.35 

2.20 

2.51 

1.52 

4.23 

2.16 

3.37 

2.04 

4.73 

2.22 

3.73 

1.68 

6.23 

2.10 

4.98 

2.27 

7.25 

1.52 

7.66 

1.33 

6.91 

1.61 

7.28 

1.52 

6.45 

2.19 

5.92 

2.36 

7.36 

1.34 

6.64 

2.03 

2.83 

1.91 

2.11 

1.46 

3.77 

2.19 

2.94 

2.03 

5.05 

2.31 

3.41 

1.75 

6.75 

1.57 

5.08 

2.36 

3.07 

1.87 

2.22 

1.27 

3.93 

2.17 

3.07 

1.96 

4.77 

2.20 

3.27 

1.49 

6.25 

1.84 

4.76 

2.24 

6.81 

1.89 

7.19 

1.99 

7.10 

1.42 

7.14 

1.72 

6.32 

2.13 

5.52 

2.30 

7.30 

1.41 

6.41 

2.09 

3.43 

2.19 

2.18 

1.39 

4.00 

2.03 

3.09 

1.95 

6.98 

2.01 

5.49 

2.46 

3.68 

1.84 

6.69 

1.56 

6.83 

1.80 

5.33 

2.37 

4.35 

2.53 

3.36 

2.29 

2.42 

1.46 

4.16 

2.34 

3.29 

2.13 

2.40 

1.46 

4.25 

2.43 

3.32 

2.21 

6.58 

2.26 

5.23 

2.45 

3.45 

1.61 

6.44 

1.83 

4.94 

2.28 

3.66 

1.72 

6.51 

2.05 

5.09 

2.36 

6.71 

1.86 

7.28 

1.63 

6.98 

1.84 

6.52 

1.74 

6.75 

1.79 

7.42 

1.58 

6.62 

1.79 

7.02 

1.73 

7.62 

1.52 

6.72 

2.11 

5.75 

2.26 

7.20 

1.31 

6.48 

1.97 

5.79 

2.24 

7.41 

1.42 

6.60 

2.04 

4.02 

2.33 

3.02 

2.12 

2.21 

1.48 

4.42 

2.32 

3.31 

2.23 

2.11 

1.38 

4.22 

2.32 

3.17 

2.17 

Table  T-2.  NASA  TLX:  MANOVA  Results. 


Position 


Low  Load 

High  Load _ 

Load 


Side 

R  Side _ 

Automation 


Position  x  Load 


Position  x  Automation _ 

Load  x  Automation 


^sition  x  Load  x  Automation 


Wilks  A  Pillai’s  Hotelling  Roy’s 
.389  0.6  Tl  1.5741  1.57 


;lling  Roy’s  F  dfl  df2  /7-level 
1.574|  1.574|  276231  6|  Toj  M6 


Table  T-3.  Mental  Demand:  ANOV A  Results. 


Position 


MS  Effect  MS  Error 
47490  l56l‘ 


utomation 


Position  X  Load 


Position  X  Automation 


Load  X  Automation _ 

Position  X  Load  X  Automation 


jdf  i  |  df  2 1  p-level 

1.261  I  13  579 


Table  T-4.  Physical  Demand:  ANOVA  Results. 


Position 

0.209 

Load 

165.639 

Automation 

3.459, 

Position  X  Load  j 


Position  X  Automation _  0.126 

Load  X  Automation  07997 


[Position  X  Load  X  Automation  0.51 1|  3.207  0.159  2  30  .853 

Table  T-5.  Temporal  Demand:  ANOVA  Results. 


utomation 

Position  X  Load  —  — 


Position  X  Automation 


Load  X  Automation 
Position  X  Load  X  Automation 


Table  T-6.  Performance:  ANOVA  Results. 


I  MS  Effect  |  MS  Error  | 


Position  X  Automation _ 

Load  X  Automation 
Position  X  Load  X  Automation 


dfT|df  2|/?-level 


.047 


.000 


.000 


1  15  .194 


Position  X  Load  _ 

Position  X  Automation _ 

Load  X  Automation 
Position  X  Load  X  Automation 


Table  T-7.  Effort:  ANOVA  Results. 


MS  Effect  MS  Error  F 


2.899 

126.090  : 


.958 


1.402  I, 
tion  0.201  1.456  0. 135 

i _ 0.492  1 .729  0.28i 

Automation  5.153  3.083  1.67^ 
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Table  T-8.  Frustration:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

jp-level 

Position 

4.047 

4.447 

WKK2ML 

1 

Load 

HEEZiS 

l 

15 

.000 

Automation 

1.748 

2 

30 

■EES 

Position  X  Load 

i 

■E 

WKM 

Position  X  Automation 

KM 

2 

■sir 

.295 

Load  X  Automation 

BEUi 

2 

Position  X  Load  X  Automation 

E 

2 
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APPENDIX  U 


PSQ  Workload  Results 


Table  U-l.  PSQ  Workload:  Means  and  Standard  Deviations. 


■■ 

■■■Hi 

H 

Controller  | 

SliapSII 

Load/Pos. 

Workload 

Question 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Collapsed 

7)  How  Hard  You  Were  Working 

7.32  2.05 

6.57  1.41 

6.54  2.36 

Aiitomationl 

7)  How  Hard  You  Were  Working 

6.57  2.96 

5.20  2.26 

5.88  2.69 

7.13  2.15 

5.57  1.90 

6.35  2.15 

6.85  2.56 

5.38  2.06 

7)  How  Hard  You  Were  Working 

6.57  2.50 

4.07  2.60 

5.32  2.81 

6.25  2.74 

5.38  2.36 

5.82  2.56 

6.41  2.59 

4.72  2.53 

7)  How  Hard  You  Were  Working 

6.82  2.50 

5.09  2.54 

5.95  2.65 

6.55  2.69 

5.84  1.96 

6.19  2.37 

6.68  2.59 

5.46  2.29 

MiYMdHI 

Table  U-l.  Working  Hard:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/7-level 

Position 

■ESH 

■EmS 

0.576 

1 

ME 

HE2 

Load 

2.502 

m 

■E 

hoe 

Automation 

HS 

Position  X  Load 

ME 

22 

Position  X  Automation 

1.185 

2 

30 

.320 

Load  X  Automation 

6.390 

m mm 

2.241 

2 

30 

.124 

Position  X  Load  X  Automation 

4.385 

4.869 

0.901 

2 

30 

.417 

Table  U-2.  Working  Hard:'  ANOVA  Results  (N=6). 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/7-level 

Position 

11.681 

7.114 

1.642 

1 

5 

.256 

Load 

mkem s 

m 

HH&a 

Automation 

mkem 

heb 

E 

ME 

Position  X  Load 

he 

3.892 

IIHUsS 

5 

.168 

Position  X  Automation 

HEiM! 

2.397 

hes&e 

10 

.994 

Load  X  Automation 

5.542 

2.625 

2.111 

2 

10 

.172 

Position  X  Load  X  Automation 

0.292 

4.308 

0.068 

2 

10 

.935 
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APPENDIX  V 


Visual  Scanning  Data  After  First  Data  Reduction  and  Analysis  Pass 
V.l  Visual  Scanning  Variables 

The  oculometer  recorded  eye  movements  during  both  practice  scenarios  and  experimental 
scenarios.  0-1  provides  a  summary  of  the  eye  movement  measures. 

Table  V-l.  Visual  Scanning  Variables 


1 .  Conditional  information  -  Aircraft  2 1 . 

2.  Conditional  information  -  Location  22. 

3 .  Conditional  information  -  Range  23 . 

4.  Conditional  information  -  Tightness  24. 

5.  Eye  motion  workload  25. 

6.  Pupil  motion  workload  26. 

7.  Visual  efficiency  27. 

8.  Mean  number  of  fixations  28. 

9.  Mean  duration  of  fixations  29. 

10.  Mean  fixation  area  30. 

1 1 .  Mean  distance  of  saccades  3 1 . 

12.  Mean  duration  of  saccades  32. 

13.  Mean  number  of  dwells  33. 

14.  Mean  dwell  area  34. 

15.  Mean  duration  of  dwells  35. 

16.  Number  of  fixations  on  target  36. 

17.  Mean  duration  of  fixations  on  target  37. 

18.  Number  of  fixations  off  target  38. 

19.  Mean  duration  of  fixations  off  target  39. 


20.  Number  of  fixations  on  radar  returns 


Mean  duration  of  fixations  on  radar  returns 

Number  of  fixations  on  data  blocks 

Mean  duration  of  fixations  on  data  blocks 

Number  of  fixations  on  other  static  objects 

Mean  duration  of  fixations  on  other  static  objects 

Number  of  fixations  on  PVD 

Mean  duration  of  fixations  on  PVD 

Number  of  fixations  on  SCRD 

Mean  duration  of  fixations  on  SCRD 

Number  of  fixations  on  map 

Mean  duration  of  fixations  on  map 

Number  of  fixations  on  flight  strips 

Mean  duration  of  fixations  on  flight  strips 

Number  of  fixations  on  keyboard 

Mean  duration  of  fixations  on  keyboard 

Number  of  fixations  on  trackball 

Mean  duration  of  fixations  on  trackball 

Number  of  fixations  on  ATWIT 

Mean  duration  of  fixations  on  ATWIT 
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V.1.1  Fixations 


A  fixation  is  a  sequence  of  at  least  6  oculometer  samples  with  an  intersample  distance  of  less 
than  1  degree  of  visual  angle.  At  1 -meter  distance,  this  corresponds  to  a  circle  with  an  8.73  mm 
radius.  The  distance  between  two  samples  is  the  norm  of  the  vectorial  difference  of  the  sample 
coordinates.  If  2  fixations  are  not  separated  by  either  a  blink  or  a  saccade  (see  definitions 
below),  these  fixations  should  be  combined  within  one  fixation.  In  summary: 

Fixation  if: 

D  =  V((xj-Xi+i)2  +(yi-yi+i)2)  >  8.73  mm 

with  D  the  distance  between  to  subsequent  samples  x  and 
y  the  horizontal  and  vertical  point  of  gaze  coordinates  in 
mm  respectively 


and: 


n  >  6 


with  n  the  number  of  samples  in  a  sequence 


and 


separated  by  a  blink  or  a  saccade 

Related  to  a  fixation  the  following  variables  need  to  be  calculated:  Fixation  Duration  and 
Fixation  Area.  Fixation  Area  is  an  approximation  of  the  area  covered  by  the  POG  due  to  eye 
movements  within  a  fixation. 

Fixation  Duration: 

FIXDUR  =  tsampie  *  Zsamples 

with  tSampie  where  the  duration  of  a  sample  ( Vgo  second) 
and  L  sample  is  the  total  number  of  samples  within  a 
fixation 


Fixation  Area: 

FIXAREA  =  (max(xflx)-min(xrlx))*(max(yfix)-min(yfix)) 

with  Xflx  and  yfix  the  sequences  of  horizontal  and  vertical 
POG  coordinates  within  a  fixation  respectively 
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Figure  V-l.  Example  of  Fixation  Distribution.  The  units  for  horizontal  and  vertical  coordinates 
are  in  pixels.  The  top  left  comer  corresponds  with  the  top  left  comer  of  the  radar  display. 

o 

-200 
400 
-600 
-800 
-1000 
-1200 
-1400 
-1600 
-1800 
-2000 

0  500  1000  1500  2000 

Figure  V-2.  Example  of  Fixation  Distribution.  The  flight  path  of  a  departure,  BTA3721  is 
superimposed.  The  circles  represent  fixations  that  were  identified  as  fixation  on  flight 
BTA3721 .  The  units  for  horizontal  and  vertical  coordinates  are  in  pixels 


J _ _ I - L 


V-3 


\ 

<**'*'■> 

"■*a&  '» 


pyx.* 


-••'■'l  .H'iu  ?:<.*”*] 

I 

I 


;.,r  fUftlip 
j»&K  ‘  I. 

'r 

1 


tMUVUZttf 
. 1 

"  x.  / 

/V 


,  .i1!  I — i 


ixr.wai: 


jia^T/pT*/  j”" 

i'"*’fiau*fSt4»''  .  ,v.“  ^TJSw? 


v  t  rrattfli-a 


P**  \ 

/ 


1  ,?i will**?4?* 


'  T'V  ,  . 

i  nv' 


^tf^SPMrn 


h~* 


.JUJIIJ 


p»K»;v.  *.  •*'  i 

LJ  l'  LJ 


XZUK.fi  o  "'•*. 

Lj 


^74 


Figure  V-3.  Example  of  point  of  one  second  of  point  of  gaze  data  integrated  with  simulator  data 

on  position  of  radar  returns  and  data  blocks. 


V.1.2  Blink 

A  blink  is  the  complete  or  partial  closure  of  the  eye.  The  oculometer  will  suggest  that  the 
velocity  at  the  start  and  end  of  a  blink  was  greater  than  700  degrees  per  second,  which 
corresponds  with  6.108  m/s.  This  is  physically  impossible,  but  it  does  give  us  a  way  to  determine 
start  and  end  of  a  blink.  A  blink  starts  after  the  last  sample  of  the  previous  fixation  and  stops 
before  the  first  sample  of  the  next  fixation.  In  summary: 

Blink  if: 


VEL  =  V((xi-xi+i)2 +(yi-yi+i)2)  /  tsampie  >  6.108  m/s 

with  VEL  being  the  a  crude  estimate  of  the  tangential 
velocity  and  x  and  y  the  horizontal  and  vertical  point  of 
gaze  coordinates  in  mm  respectively.  The  index  denotes 
the  current  sample  i  and  next  sample  i+1  respectively 

and: 

n  >12  with  n  the  number  of  samples  in  a  sequence 
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Related  to  a  blink  the  following  variables  need  to  be  calculated:  Fixation  Duration  and  Blink 
Distance.  Blink  Distance  is  the  distance  covered  by  the  POG  due  to  eye  movements  during  a 
blink. 

Blink  Duration: 


BLNKDUR  =  tSampie  *  Esamples 

with  tsamp[e  where  the  duration  of  a  sample  ('/go  second) 
and  Esample  is  the  total  number  of  samples  within  a  blink 

Blink  Distance: 

BLNKDST  -  (xn-xp)*(y„-yp) 

with  x  and  y  the  horizontal  and  vertical  point  of  gaze 
coordinates  in  mm  respectively.  The  index  denotes  the 
last  sample  of  the  previous  fixation  p  and  first  sample  of 
the  next  fixation  n  respectively 


V.1.3  Saccade 

A  saccade  is  the  ballistic  movement  of  the  eye  from  one  fixation  to  the  next.  A  saccade  is 
characterized  by  fast  eye  movements  of  up  to  700  degrees  per  second.  The  cut-off  for  a  saccade 
is  a  difference  in  distance  between  two  subsequent  saccades  that  is  greater  or  equal  to  8.73  mm, 
lasts  at  least  3  samples  (or  a  velocity  of  0.524  m/s),  and  the  velocity  is  less  or  equal  to  700  degrees 
per  second  (6. 108  m/s).  The  saccade  will  start  at  the  end  of  the  last  sample  of  the  previous 
fixation  and  will  end  at  the  beginning  of  the  first  sample  of  the  next  fixation.  In  summary: 

0.524  >VEL>  6.108  m/s 


and: 


n  >  2 

Related  to  saccades  a  number  of  variables  need  to  be  calculated:  Saccade  Duration,  Saccade 
Distance,  and  Saccade  Velocity.  The  saccade  distance  is  the  angular  distance  traveled  during  a 
saccade  in  degrees.  The  saccade  velocity  is  the  average  velocity  within  a  saccade  in  degrees  per 
second. 

Saccade  Duration: 


SACDUR=  tsampie  *  Esamples 

with  tsampie  where  the  duration  of  a  sample  (*/6o  second) 
and  Esample  is  the  total  number  of  samples  within  a 
saccade 
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Saccade  Distance: 


SACDST  =  (xn-xp)*(yn-yp) 

with  x  and  y  the  horizontal  and  vertical  point  of  gaze 
coordinates  in  mm  respectively.  The  index  denotes  the 
last  sample  of  the  previous  fixation  p  and  first  sample  of 
the  next  fixation  n  respectively 


Saccade  Velocity: 


SACVEL  =  Z  (V((xi-xi+i)2  +(yi-yi+i)2))  /  tsamp,e  *  nsaccade 

with  Sample  where  the  duration  of  a  sample  ('/6o  second) 
and  nsaccade  is  the  number  of  samples  within  the  saccade 


V.1.4  Dwell 

A  dwell  is  defined  as  a  sequence  of  fixations  that  return  to  a  location  within  1  degree  of  visual 
from  a  target  location  or  within  1  degree  of  visual  angle  if  the  POG  does  not  rest  on  a  target. 
This  way  included  in  a  dwell  are  also  moving  targets. 

Related  to  dwells  a  number  of  variables  need  to  be  calculated:  Dwell  Duration  and  Dwell  Area. 
Dwell  Duration  is  the  duration  between  the  start  of  the  first  sample  of  the  first  fixation  and  the 
end  of  the  last  sample  of  the  last  fixation  within  a  dwell  sequence.  Dwell  Area  is  an 
approximation  of  the  area  covered  by  the  POG  within  a  dwell. 

Dwell  Duration: 

DDUR  —  tn  fiX  ni  -  t^fix  I 


with  ti,fix  i  is  the  start  of  the  first  sample  of  the  first 
fixation  and  tn,f,x  m  is  the  end  (sample  n)  of  the  last 
fixation  (fixation  m). 


Dwell  Area: 

DAREA  =  (max(xflx)-min(Xf,x))*(max(yf1x)-min(yflx)) 

with  Xflx  and  yf,x  the  sequences  of  horizontal  and  vertical 
POG  coordinates  within  a  dwell  respectively 
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V.1.5  Visual  Efficiency 

Visual  efficiency  is  defined  as  the  proportion  of  the  total  scanning  time  that  is  spent  fixating. 
Visual  Efficiency: 

VISEFF  =  (mean(FIXDUR)  *  Nfix)  / 

(mean(FIXDUR)  *  Nfix  +  mean(SACDUR)  *  Nsac) 

In  fact,  this  is  nothing  more  than  the  portion  of  the  time  that  the  eye  is  fixed  once  the  blinks  are 
removed: 

Visual  Efficiency: 

VISEFF  =  ZFIXDUR  /  (ZFIXDUR  +  ZSACDUR) 

with  ZFIXDUR  the  sum  of  the  duration  of  the  fixations, 
2SACDUR  the  sum  of  the  duration  of  the  saccades  and 
TIME  the  total  time  in  seconds. 


V.1.6  Eye  Motion  Workload 

Eye  Motion  Workload  is  defined  as  the  average  saccade  motion  in  degrees  by  the  number  of 
saccades,  or: 

Eye  Motion  Workload: 

EYEMWL  =  mean  (SACDST)  *  Nsac  /  TIME 

with  Nsac  the  number  of  saccades  within  the  interval 
under  study  and  TIME  the  total  time  in  seconds. 

In  fact,  this  is  nothing  more  than  the  total  distance  traveled  divided  by  the  total  the  time: 

Eye  Motion  Workload: 

EYEMWL  =  2SACDST  /  TIME 

with  ZSACDST  the  sum  of  the  distance  of  the  saccades 
in  degrees  and  TIME  the  total  time  in  seconds. 
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V.1.7  Pupil  Motion  Workload 

Pupil  Motion  Workload  is  defined  as  the  sum  of  the  average  pupil  diameter  within  a  fixation 
divided  by  the  total  time  within  the  interval  under  consideration. 

Pupil  Motion  Workload 

PUPMWL  =  Z||mean(PUPDIAM)rlx  mean(PUPDIAM)fiX  i+I)||  /  TIME 

with  PUPDIAM  the  pupil  diameter  in  mm  based  on  a 
conversion  from  ASL  arbitrary  units  to  mm  of  0.044  mm 
per  ASL  unit.  The  index  fix  i  and  fix  i+1  denote  the  i-th 
and  the  i+lth  fixation  respectively 

It  seems  if  the  author  of  the  article  that  this  measure  was  based  on  was  after  the  “distance” 
traveled  during  an  interval.  I  is  of  course  possible  to  separate  the  oculometer  samples  that  do  not 
include  blinks  and  then  to  calculate  the  cumulative  sum  of  the  pupil  diameter  differences.  This 
may  be  a  more  accurate  estimate  of  pupil  workload: 

Pupil  Average  Work: 

for  fixations  or  saccades: 

PUP  AW  =  Z||PUPDIAMi  -  PUPDIAMl+i|| 

with  i  and  i+1  oculometer  sample  i  an  i+1  respectively. 

In  this  case  the  oculometer  samples  that  occur  during 
blinks  are  removed  from  the  time  series  of  data. 


V.  1.8  Conditional  Information 

The  conditional  information  is  defined  by  Brillouin  (1962)  as  described  in  Ellis  (1986).  The 
formula  will  here  be  given  without  getting  too  much  into  the  details: 

CONINF  =  2  pi  *  [£  py  *  log2  (p-,j)]  with  i  *j 

with  pi  is  simple  probability  of  viewing  target  i,  and  py  is 
the  probability  of  a  transition  from  target  i  to  target  j. 
Simple  probability  was  defined  by  Ellis  (1986)  as  the 
percentage  of  time  spent  on  each  particular  target  or 
jumping  between  each  target.  Here  we  will  calculate  it 
not  as  a  percentage  of  time,  but  the  ratio  of  the  number  of 
times  on  a  target  and  the  total  number  of  fixations  and  the 
number  of  transitions  and  the  total  number  of  saccades 
for  pi  and  pij  respectively. 
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APPENDIX  W 


Visual  Scanning 


Table  W-l.  Fixations:  Means  and  Standard  Deviations. 


Fixations 

Load 

if 

H 

nipt 

W  lllQOW 

Means 

SDs 

Means 

SDs 

Means  | 

SDs 

v; 

Number 

2976.08 

309.37 

3046.82 

208.26 

3011.45 

261.90 

Mean  Duration 

435.58 

63.87 

428.99 

62.61 

432.28 

62.31 

Stdev  Duration 

237.98 

126.30 

210.71 

70.43 

224.34 

101.54 

Mean  Area 

0.96 

0.35 

0.99 

0.29 

0.98 

0.32 

!  Ipi  | 

Stdev  Area 

2.45 

1.71 

2.86 

1.61 

2.66 

1.65 

Efficiency 

0.76 

0.05 

0.75 

0.08 

0.76 

0.06 

-■  *  -  ^ 

Number 

2873.96 

240.67 

2968.88 

297.09 

2921.42 

270.30 

a 

Mean  Duration 

490.29 

51.44 

453.95 

74.80 

472.12 

65.79 

ISillp 

Stdev  Duration 

326.01 

88.36 

288.16 

152.84 

307.09 

124.30 

r?”iSv 

Mean  Area 

1.12 

0.33 

0.96 

0.25 

1.04 

0.30 

jjlpfjl 

Stdev  Area 

3.27 

2.16 

2.50 

1.35 

2.89 

1.81 

Efficiency 

0.81 

0.04 

0.75 

0.08 

0.78 

0.07 

Number 

2851.32 

336.75 

2963.58 

255.52 

2907.45 

299.52 

gL,:;.  i; 

Mean  Duration 

498.55 

61.68 

479.46 

43.34 

489.00 

53.33 

Stdev  Duration 

328.52 

105.91 

286.50 

65.96 

307.51 

89.38 

'  r  -:i:  ip: 

Mean  Area 

0.97 

0.24 

0.92 

0.25 

0.95 

0.24 

splili 

Stdev  Area 

2.46 

1.17 

2.13 

1.03 

2.29 

1.10 

Efficiency 

0.81 

0.04 

0.79 

0.04 

0.80 

0.04 

. < 

Number 

2900.45 

297.04 

2993.09 

253.63 

2946.77 

278.65 

Mean  Duration 

474.81 

64.48 

454.13 

63.79 

464.47 

64.64 

Stdev  Duration 

297.50 

113.90 

261.79 

108.44 

279.65 

112.06 

:'jBp£ 

Mean  Area 

1.02 

0.31 

0.96 

0.26 

0.99 

0.29 

!'f'^fv'wv 

Stdev  Area 

2.73 

1.74 

2.50 

1.35 

2.61 

1.55 

rr- 1 

Efficiency 

0.79 

0.05 

0.77 

0.07 

0.78 

0.06 
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Table  W-2.  Saccades:  Means  and  Standard  Deviations. 


Saccades 


Window 


Number 
Mean  Duration 
Stdev  Duration 
Mean  Distance 
Stdev  Distance 
Eye 


Number 
Mean  Duration 
Stdev  Duration 
Mean  Distance 
Stdev  Distance 
Eye 


Number 
Mean  Duration 
Stdev  Duration 
Mean  Distance 
Stdev  Distance 
Eye 


Number 
Mean  Duration 
Stdev  Duration 
Mean  Distance 
Stdev  Distance 
Eye 


Means  | 

SDs 

iSRHSr 

Bl 

Means 

SDs 

2637.26 

275.24 

2711.22 

342.83 

2674.24 

308.12 

148.27 

24.25 

154.00 

37.57 

151.13 

31.24 

72.19 

44.90 

64.65 

32.26 

68.42 

38.65 

4.71 

0.76 

4.56 

0.44 

4.63 

0.61 

55.39 

23.33 

52.85 

16.17 

54.12 

19.79 

5.91 

0.83 

5.86 

0.90 

5.89 

0.85 

2550.01 

288.01 

2701.53 

379.21 

2625.77 

340.06 

132.38 

22.07 

161.71 

54.91 

147.04 

43.78 

54.88 

44.23 

127.54 

288.11 

91.21 

206.09 

4.45 

0.50 

4.50 

0.56 

4.48 

0.52 

41.57 

15.45 

42.84 

12.60 

42.20 

13.88 

5.34 

0.65 

5.63 

0.67 

5.49 

0.67 

2600.28 

502.89 

2785.95 

322.79 

2693.11 

426.24 

130.87 

17.42 

136.86 

26.90 

133.87 

22.50 

53.56 

30.52 

45.01 

23.03 

49.28 

26.95 

4.22 

0.41 

4.29 

0.44 

4.26 

0.42 

52.58 

75.17 

37.37 

12.71 

44.98 

53.59 

5.16 

0.86 

5.62 

0.67 

5.39 

0.79 

2595.85 

364.24 

2732.90 

343.67 

2664.37 

358.91 

137.17 

22.43 

150.86 

41.88 

144.01 

34.12 

60.21 

40.48 

79.07 

168.10 

69.64 

121.99 

4.46 

0.60 

4.45 

0.49 

4.45 

0.54 

49.85 

45.71 

44.35 

15.09 

47.10 

33.97 

5.47 

0.84 

5.70 

0.75 

5.59 

0.80 

Table  W-3.  Blinks:  Means  and  Standard  Deviations. 


Blinks 

Window 

SSliil1 

OiMli* 

Means 

SDs 

Means 

SDs 

Means 

SDs 

Number 

306.31 

144.19 

273.65 

170.83 

289.98 

156.38 

Mean  Duration 

471.57 

536.47 

543.22 

601.34 

507.39 

561.74 

** 

Stdev  Duration 

292.26 

549.73 

375.23 

611.32 

333.75 

573.44 

Mean  Distance 

14.85 

3.44 

15.40 

5.16 

15.12 

4.32 

sciii 

Stdev  Distance 

192.36 

179.56 

129.34 

86.58 

160.85 

142.31 

Number 

305.94 

190.65 

269.96 

135.31 

287.95 

163.65 

S|» 

Mean  Duration 

503.67 

523.39 

547.06 

597.49 

525.37 

552.97 

}J|f| 

Stdev  Duration 

391.57 

544.47 

412.75 

608.88 

402.16 

568.29 

s  ?  jgjfs 

Mean  Distance 

15.77 

3.41 

16.10 

3.55 

15.93 

3.42 

lf||  jj 

Stdev  Distance 

186.53 

148.52 

125.33 

48.21 

155.93 

112.98 

Number 

239.02 

171.25 

268.13 

167.68 

253.57 

167.38 

L;:Jp;2 

Mean  Duration 

569.41 

588.39 

493.50 

554.11 

531.46 

563.53 

Ul¥: 

Stdev  Duration 

412.30 

604.91 

338.17 

557.17 

375.24 

573.31 

■ijii 

Mean  Distance 

14.59 

4.82 

14.36 

4.90 

14.48 

4.78 

m 

Stdev  Distance 

378.64 

868.51 

136.63 

67.21 

257.63 

618.30 

i  ^gpi 

Number 

283.76 

169.17 

270.58 

155.36 

277.17 

161.69 

-;£vV|LiJ  - 

Mean  Duration 

514.88 

539.87 

527.93 

572.66 

521.41 

553.61 

|'f||  ’ 

Stdev  Duration 

365.38 

557.35 

375.38 

581.05 

370.38 

566.34 

71  W.  ' 

Mean  Distance 

15.07 

3.89 

15.29 

4.55 

15.18 

4.21 

4 

Stdev  Distance 

252.51 

515.94 

130.43 

67.81 

191.47 

371.13 

Table  W-4.  Scene  Based  Fixations:  Means  and  Standard  Deviations. 


y  ft? 

IPT?!555i7 

WSc 

-  -5/V ;  * 

siv. 

~  \£ 

111  f* 

SSH 

Ill  | 

Display/Load 

SBF 

Window 

Low  Load 

High  Load 

Load  Collapsed 

Low  Load 

High  Load 

Load  Collapsed 

Low  Load 

High  Load 

Collapsed 

Means 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  ] 

SDs 

Means  | 

SDs 

Means  | 

SDs 

§ggl|| 

Number 

381.90 

365.00 

470.93 

406.53 

426.42 

382.72 

1525.65 

458.01 

1501.75 

473.55 

1513.70 

458.43 

953.78 

709.62 

986.34 

680.21 

970.06 

689.73 

dig? 

Percent 

12.90 

12.49 

15.90 

14.08 

14.40 

13.18 

51.66 

15.68 

49.42 

15.46 

50.54 

15.36 

32.28 

24.12 

32.66 

22.39 

32.47 

23.09 

is® 

Mean  Duration 

384.08 

76.53 

394.58 

92.82 

389.33 

83.85 

505.81 

67.65 

500.26 

75.43 

503.04 

70.54 

444.95 

94.19 

447.42 

99.02 

446.18 

95.87 

Ifill 

Number 

1011.97 

379.44 

1069.25 

391.99 

1040.61 

380.61 

1212.40 

258.27 

1258.43 

482.67 

1235.42 

381.51 

1112.18 

335.12 

1163.84 

1138.01 

390.56 

||l| 

Percent 

34.99 

12.19 

36.78 

15.91 

35.88 

13.97 

42.38 

8.66 

41.83 

14.72 

42.10 

11.88 

38.68 

11.06 

39.30 

15.29 

38.99 

13.24 

Mean  Duration 

533.86 

86.69 

499.55 

101.03 

516.71 

94.23 

515.14 

58.41 

473.19 

89.33 

494.16 

77.24 

73.33 

486.37 

94.76 

505.44 

86.22 

iggp 

Number 

1538.31 

526.88 

1444.03 

395.33 

1491.17 

460.70 

875.43 

334.22 

999.72 

386.93 

937.57 

361.22 

1206.87 

549.34 

1221.87 

446.10 

1214.37 

496.46 

I 

Percent 

54.04 

16.60 

49.35 

15.02 

51.70 

15.76 

30.69 

11.25 

33.29 

11.50 

31.99 

11.26 

42.37 

18.31 

41.32 

15.48 

41.84 

16.83 

0  < 

Mean  Duration 

534.91 

74.25 

523.20 

58.83 

529.06 

66.17 

500.77 

51.56 

495.51 

50.73 

498.14 

50.39 

517.84 

65.23 

509.35 

55.84 

513.60 

60.39 

sgfffggsgj 

■  -  k  \ 

Number 

977.39 

636.63 

994.74 

561.85 

986.06 

597.30 

1204.50 

442.59 

1253.30 

486.43 

1228.90 

463.22 

1090.94 

557.18 

1124.02 

538.63 

1107.48 

546.80 

pit 

Percent 

33.98 

21.77 

34.01 

20.26 

33.99 

20.92 

41.58 

14.76 

41.51 

15.23 

41.54 

14.92 

37.78 

18.89 

37.76 

18.22 

37.77 

18.51 

p'tl 

Mean  Duration 

484.29 

105.62 

472.44 

101.50 

478.36 

103.21 

507.24 

58.60 

489.65 

72.98 

498.45 

66.43 

495.76 

85.74 

481.05 

88.36 

488.41 

87.14 
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Table  W-5.  Scene  Based  Saccades:  Means  and  Standard  Deviations. 


SB  S 

Window 

gagaKi 

SM 

„oad  " 

Load  Collapsed 

'  Low  Load 

■■ 

Low  Load 

Display /Load 
Collapsed 

,._L  _ 

SDs 

KBl 

SDs 

SDs 

SDs 

Means  | 

EES 

SDs 

Means  [ 

m'im 

1 1 

Number 

343.21 

315.67 

413.49 

365.97 

378.35 

338.08 

1270.77 

416.69 

1233.61 

452.67 

1252.19 

428.40 

806.99 

595.20 

823.55 

580.98 

815.27 

583.51 

Percent 

13.01 

11.88 

15.54 

13.93 

14.28 

12.80 

48.60 

16.43 

45.67 

15.55 

47.14 

15.80 

30.80 

22.93 

30.61 

21.10 

30.71 

3.42 

21.86 

0.80 

EH? 

Mean  Distance 

4.25 

0.98 

3.44 

0.57 

3.85 

0.89 

3.11 

0.33 

2.87 

0.36 

2.99 

0.36 

3.68 

0.93 

3.16 

0.55 

pi 

Number 

Percent 

952.52 

37.23 

337.77 

991.92 

392.72 

972.22 

360.88 

971.90 

266.54 

1023.11 

448.82 

997.50 

364.04 

962.21 

299.46 

1007.52 

415.15 

984.86 

359.80 

‘  1 

12.08 

37.15 

15.52 

37.19 

13.68 

38.20 

9.23 

37.53 

15.23 

37.87 

12.39 

37.72 

10.59 

37.34 

15.13 

0.50 

37.53 

12  95 

Mean  Distance 

335 

037 

3.56 

0.48 

3.46 

0.43 

3.18 

0.38 

2.97 

0.31 

3.07 

0.36 

3.26 

0.38 

3.26 

3.26 

0  44 

d| 

Number 

Percent 

1487.80 

61 1.09 

1409.40 

409.86 

1448.60 

513.39 

689.42 

303.85 

822.15 

349.58 

755.78 

329.17 

1088.61 

624.39 

1115.77 

478.97 

1102.19 

552.19 

[s  *1; 

56.82 

16.99 

50.94 

14.81 

53.88 

15.96 

26.92 

11.09 

29.21 

11.15 

28.06 

11.00 

41.87 

20.73 

40.07 

16.98 

0.45 

40.97 

3.24 

1 8.82 

BEg 

Mean  Distance 

3.53 

0.40 

3.49 

0.25 

3.51 

0.33 

3.13 

0.25 

2.82 

0.34 

!  2.98 

0.33 

3.33 

0.39 

3.15 

0.43 

R9H 

Number 

Percent 

927.84 

35.69 

640.86 

938.27 

562.03 

933.05 

599.57 

977.36 

406.32 

1026.29 

444.42 

1001.83 

424.27 

952.60 

534.31 

982.28 

505.91 

967.44 

519.15 

Ujy 

22.62 

34.55 

20.63 

35.12 

21.54 

37.91 

15.25 

37.47 

15.39 

37.69 

15.24 

36.80 

19.22 

36.01 

18.16 

36.40 

3.31 

18.66 

0.58 

Mean  Distance 

3.71 

0.75 

3.50 

0.44 

3.60 

0.62 

3.14 

0.32 

2.89 

0.34 

3.01 

0.35 

3.43 

0.64 

3.19 

0.50 

Table  W-6.  Object  Based:  Means  and  Standard  Deviations. 


High  Load 


Load  Coll 


Means 


Low  Load 


High  Load 


Means  SDs 


|  Window  Low  Load  High  Load  Load  Coll.  Low  Load 
[Means |  SDs  Means |  SDs  Means]  SDs  Means 


892.16  240.29  922. 

541.18  73.97  528. 

Number  725.97  243.93  776.02  312.49 

Mean  Duration  554,28  59.99  500.38  97.96i  ja  /.jj  oh.hoi  o/j.qu  /a.yiioao.iu  /b.2U|&l4U4  91, g 

Number  500.96  224.96  597.47  217.24  549.22  223.00  260.04  103.99  329.35  140.23  294.70  126.45  380.50  21L42  463.41  225  62 

Mean  Duration  531,70  65.61519.42  58.99  525.56  61.69  703.75  81.87  701.29  62.37  702.52  71.60  617.72  113.87  610.36  110  01 

Number  706.36  282.51  765.32  284.64  735.84  283.63  333.52  121.28  394.27  169.94  363.89  149.99^9.94  286.15  579.79  298  58 

Mean  Duration  542.39  66.00  515.94  78.97|529.16  73.6o|682.36  86.08[688.52  70.51  685.44  78.33  612.37  103.78  602.23  11432 


443.13  160.25  654.91  307.32 

672.84  85.91  605.35  10731 

353.85  125.48  524.41  269.98 
680.95  76.20  614.04  91  S“ 


Info/Load 

Collapsed 


Means  SDs 


675.22  306.14 
603.72  106.06 
552.42  292.47 
604,14  111.18 
421.96  220.88 
614.04  111.13 
549.87  293.21 
607.30  109.01 


Table  W-7.  Structure:  Means  and  Standard  Deviations. 


Structure 

Window 

\mm 

:0ip3 

sunups 

Means 

SDs 

Means 

SDs 

Means  SDs 

Object-Based 

0.13 

0.05 

0.07 

(■si 

Range-Based 

1.00 

0.01 

1.00 

|l|i  Jp 

Box-Based 

4.24 

0.17 

4.14 

H  I  1 1 

Ring-Based 

0.72 

0.04 

0.71 

1 

Object-Based 

0.12 

0.04 

0.08 

0.02 

0.10  0.04 

n  1 

Range-Based 

1.00 

0.01 

1.00 

0.02 

1.00  0.02 

j  i  |  in 

Box-Based 

4.21 

0.14 

4.10 

0.23 

4.16  0.19 

1  ll  1 

Ring-Based 

0.71 

0.06 

0.70 

0.06 

0.71  0.06 

Object-Based 

0.12 

0.04 

0.07 

0.02 

0.10  0.04 

Range-Based 

1.00 

0.02 

1.00 

0.02 

1.00  0.02 

Box-Based 

4.09 

0.24 

4.00 

0.25 

4.04  0.24 

Ring-Based 

0.69 

0.06 

0.70 

0.03 

0.69  0.05 

•O 

Object-Based 

0.12 

0.04 

0.07 

0.02 

■BBSS 

' «  •  t 

Range-Based 

1.00 

0.01 

1.00 

0.02 

■B 

IJggpg 

Box-Based 

4.18 

0.19 

4.08 

0.21 

Ring-Based 

0.71 

0.06 

0.70 

0.04 

■BBS 

Table  W-8.  Limited  Data  Block:  Means  and  Standard  Deviations. 


LDB 

Window 

8 

Load  |  1 

jEcjf'Ct 

dlapsed 

Means 

SDs 

Means  | 

SDs 

Means 

SDs 

F 

Number 

22.09 

27.58 

31.61 

27.11 

26.85 

27.33 

§HB 

Percent 

0.71 

0.82 

1.02 

0.85 

0.87 

0.84 

k 

Mean  Duration 

461.95 

233.05 

441.90 

189.13 

451.92 

209.03 

8 

■fag 

Stdev  Duration 

214.17 

281.43 

200.93 

287.45 

207.55 

279.91 

h 

SaiSSKS 

Number 

16.27 

12.73 

26.86 

22.16 

21.57 

18.57 

[.- 

Percent 

0.56 

0.44 

0.88 

0.72 

0.72 

0.61 

p 

ttHl 

Mean  Duration 

349.57 

173.60 

366.81 

169.53 

358.19 

169.02 

I 

Stgi 

Stdev  Duration 

91.75 

80.31 

112.43 

89.84 

102.09 

84.47 

P 

ymK^s 

Number 

20.11 

24.51 

23.77 

23.92 

21.94 

23.90 

I 

f  :'!§§ 

Percent 

0.67 

0.78 

0.78 

0.79 

0.72 

0.77 

f 

Mean  Duration 

381.72 

201.14 

361.95 

215.79 

371.83 

205.45 

1 

Jr  a# 

Stdev  Duration 

110.84 

89.78 

161.33 

354.50 

136.09 

255.67 

Number 

19.49 

22.19 

27.42 

24.17 

23.45 

23.42 

fe.SPi 

Percent 

0.65 

0.69 

0.90 

0.78 

0.77 

0.74 

Mean  Duration 

397.75 

205.29 

390.22 

191.88 

393.98 

197.69 

| 

5 . HI 

Stdev  Duration 

138.92 

181.28 

158.23 

265.32 

148.58 

226.22 
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Table  W-9.  Fixation  Number:  Means  and  Standard  Deviations. 


Number 

Window 

Means  ' 

SDs 

Means 

SDs 

Means  1  SDs  ^ 

D-side  Comm.  Panel 

2.63 

2.89 

3.96 

6.25 

3.29 

4.84 

R-side  Comm.  Panel 

3.22 

4.81 

2.42 

3.28 

2.82 

4.07 

m* 

D-side  Map 

8.61 

8.66 

7.79 

5.89 

8.20 

7.30 

'  O  '  : 

•  P»,:  f-.i  -J./J' 

'  +*>  ■  • 

R-side  Map 

7.28 

10.14 

6.89 

9.05 

7.08 

9.45 

Keyboard  Area 

216.11 

133.17 

330.49 

162.00 

273.30 

157.03 

CRD 

14.41 

16.65 

20.88 

21.77 

17.65 

19.35 

r 

ATWIT 

19.21 

11.63 

24.38 

13.50 

21.80 

12.67 

l  ■  r 

Flight  Strip  Bay  1 

33.54 

44.19 

39.85 

58.11 

36.70 

50.88 

Flight  Strip  Bay  2 

763.52 

493.87 

637.47 

401.74 

700.50 

447.46 

i6 :J 

D-side  Comm.  Panel 

4.69 

6.75 

4.46 

4.33 

4.58 

5.58 

R-side  Comm.  Panel 

2.60 

5.33 

0.61 

0.97 

1.60 

3.90 

D-side  Map 

28.05 

47.37 

9.86 

9.02 

18.95 

34.79 

r:J  ■ 

R-side  Map 

18.46 

38.88 

7.08 

9.60 

12.77 

28.45 

'::W 

-  S  ■  ’ 

Keyboard  Area 

189.17 

115.96 

227.93 

138.22 

208.55 

127.04 

I  : 

CRD 

17.29 

10.42 

14.63 

14.02 

15.96 

12.22 

i 

ATWIT 

20.90 

10.57 

23.38 

8.78 

22.14 

9.64 

|J1 

Flight  Strip  Bay  1 

10.04 

8.93 

25.60 

69.21 

17.82 

49.18 

Flight  Strip  Bay  2 

358.40 

157.00 

327.66 

148.80 

343.03 

151.28 

:i;-J -*'-5 

D-side  Comm.  Panel 

10.77 

17.94 

11.57 

23.28 

11.17 

20.45 

R-side  Comm.  Panel 

3.23 

5.51 

1.04 

1.81 

2.13 

4.18 

i-  ffj  >4:=-;-; 

D-side  Map 

19.98 

20.00 

9.24 

9.05 

14.61 

16.21 

&&!;  .  flfr:*:v^lS 

.  e  Tr-}:;? 

0 

R-side  Map 

10.20 

26.36 

2.90 

3.56 

6.55 

18.87 

Keyboard  Area 

114.99 

76.47 

206.67 

124.24 

160.83 

111.66 

Ife  ^:0f:  ‘  - 

CRD 

11.51 

10.64 

8.48 

6.63 

9.99 

8.85 

P«B 

ATWIT 

21.63 

6.85 

21.21 

11.67 

21.42 

9.42 

■ 

Flight  Strip  Bay  1 

10.35 

10.14 

11.98 

15.47 

11.16 

12.90 

Flight  Strip  Bay  2 

234.91 

143.51 

246.77 

127.18 

240.84 

133.52 

1§S11 

D-side  Comm.  Panel 

6.03 

11.49 

6.66 

14.27 

6.35 

12.89 

^:;h- v,.:cfr;.L:  ,;._ 

R-side  Comm.  Panel 

3.02 

5.12 

1.36 

2.32 

2.19 

4.04 

Ilv-;. 

D-side  Map 

18.88 

30.54 

8.96 

8.00 

13.92 

22.76 

W  . 

R-side  Map 

11.98 

27.57 

5.62 

7.96 

8.80 

20.43 

:■  .=  :  •r:i: 

|  '  |  '  • 

Keyboard  Area 

173.42 

116.98 

255.03 

149.62 

214.23 

139.74 

'  1 

CRD 

14.40 

12.84 

14.66 

15.94 

14.53 

14.40 

is  | 

ATWIT 

20.58 

9.74 

22.99 

11.31 

21.79 

10.57 

:.  5  ;: 

Flight  Strip  Bay  1 

17.98 

28.37 

25.81 

53.06 

21.89 

42.50 

Flight  Strip  Bay  2 

452.28 

379.93 

403.97 

304.46 

428.12 

343.31 

W-6 


Table  W-10.  Fixation  Percent:  Means  and  Standard  Deviations. 


Percent 


Window 


D-side  Comm.  Panel 
R-side  Comm.  Panel 
D-side  Map 
R-side  Map 
Keyboard  Area 
CRD 
ATWIT 

Flight  Strip  Bay  1 
Flight  Strip  Bay  2 


D-side  Comm.  Panel 
R-side  Comm.  Panel 
D-side  Map 
R-side  Map 
Keyboard  Area 
CRD 
ATWIT 

Flight  Strip  Bay  1 
Flight  Strip  Bay  2 


D-side  Comm.  Panel 
R-side  Comm.  Panel 
D-side  Map 
R-side  Map 
Keyboard  Area 
CRD 
ATWIT 

Flight  Strip  Bay  1 
Flight  Strip  Bay  2 


D-side  Comm.  Panel 
R-side  Comm.  Panel 
D-side  Map 
R-side  Map 
Keyboard  Area 
CRD 
ATWIT 

Flight  Strip  Bay  1 
Flight  Strip  Bay  2 


Means 


0.09 

0.11 

0.29 

0.25 

7.24 

0.47 

0.65 

1.07 

25.28 


0.16 

0.09 

0.94 

0.66 

6.55 

0.60 

0.72 

0.35 

12.55 


0.38 

0.11 

0.69 

0.33 

3.99 

0.39 

0.78 

0.36 

8.24 


0.21 

0.10 

0.64 

0.41 

5.93 

0.49 

0.72 

0.59 

15.36 


0.10 

0.18 

0.29 

0.38 

4.56 

0.49 

0.40 

1.36 

15.43 


0.22 

0.19 

1.53 

1.40 

4.03 

0.36 

0.37 

0.32 

5.68 


0.63 

0.17 

0.64 

0.80 

2.50 

0.36 

0.30 

0.35 

4.83 


0.40 

0.18 

0.99 

0.95 

3.97 

0.41 

0.36 

0.88 

12.13 


0.02 

0.35 

0.23 

7.50 

0.49 

0.80 

0.81 

11.03 


0.37 

0.04 

0.32 

0.10 

6.82 

0.28 

0.74 

0.43 

8.26 


0.22 

0.04 

0.31 

0.19 

8.31 

0.49 

0.78 

0.83 

13.29 


Means 


0.11 

0.09 

0.27 

0.24 

8.93 

0.58 

0.73 

1.16 

22.93 


0.16 

0.06 

0.65 

0.45 

7.02 

0.55 

0.76 

0.58 

11.79 


0.37 

0.07 

0.51 

0.21 

5.41 

0.34 

0.76 

0.40 

8.25 


0.30 

7.12 

0.49 

0.75 

0.71 

14.32 


SDs 


0.16 

0.14 

0.24 

0.34 

5.01 

0.64 

0.43 

1.59 

14.00 


0.19 

0.14 

1.13 

1.02 

4.09 

0.41 

0.34 

1.53 

5.42 


0.66 

0.13 

0.53 

0.58 

3.53 

0.30 

0.39 

0.50 

4.47 


'.42 
'.14 
i.74 
0.71 
4.45 
0.48 
0.38 
1.34 
10.94 
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Table  W-ll.  Fixation  Duration:  Means  and  Standard  Deviations. 


Duration 

Window 

Means 

SDs 

Means 

Means  | 

31  Fnlcral  1 

SDs 

D-side  Comm.  Panel 

116.75 

126.13 

74.23 

93.06 

95.49 

111.15 

«r. 

R-side  Comm.  Panel 

99.82 

113.75 

81.62 

77.96 

90.72 

96.37 

D-side  Map 

195.35 

123.15 

161.91 

92.36 

178.63 

108.42 

R-side  Map 

177.53 

140.52 

257.13 

459.17 

217.33 

336.46 

g* 

c  ,g  '■ 

Keyboard  Area 

259.32 

68.79 

276.88 

68.13 

268.10 

67.93 

«?."•  -  3.  ■  '•  vv-' 
rV  '  3 

CRD 

306.92 

192.47 

309.33 

131.87 

308.12 

162.29 

: 

ATWIT 

327.88 

230.50 

293.47 

108.43 

310.67 

178.05 

v- ; ; 

Flight  Strip  Bay  1 

199.13 

73.08 

181.87 

105.83 

190.50 

89.89 

fel+ni  U*  ;;  *C -•  KUtwii.w 

Flight  Strip  Bay  2 

321.40 

42.40 

300.10 

47.46 

310.75 

45.57 

ET'.iTT'  1.: 

I 

D-side  Comm.  Panel 

176.08 

219.76 

153.99 

121.70 

165.04 

175.10 

-  ’  !i  ••  •  m  %  ’{ 

R-side  Comm.  Panel 

86.61 

88.00 

29.42 

51.90 

58.02 

76.77 

ms'  ^ 

D-side  Map 

199.73 

95.84 

221.16 

126.46 

210.45 

110.91 

F  S 

ZhJT  . 

■ 

R-side  Map 

193.21 

178.01 

149.45 

121.14 

171.33 

151.42 

%  5..  ■ 

..s- 

.«*. -t 

Keyboard  Area 

265.34 

64.86 

255.86 

35.64 

260.60 

51.70 

r  i"..  v-‘  i,  'O 

:•/  "  '>*  .  ] 

5 j 

CRD 

394.46 

185.54 

301.16 

178.77 

347.81 

185.38 

■  <  ■:  I 

•;  :  ;  ;-:M 

ATWIT 

290.61 

95.85 

348.51 

121.74 

319.56 

111.73 

r  •  •  : ■'  *  ••  ■  • 'is  J 

Flight  Strip  Bay  1 

171.59 

120.73 

152.29 

110.02 

161.94 

114.04 

iltiiais  -«  ..taii'j  .iHS  1  hiujl 

Flight  Strip  Bay  2 

338.77 

57.78 

302.70 

31.43 

320.74 

49.28 

fr-  ■ 

D-side  Comm.  Panel 

143.42 

82.94 

129.49 

128.02 

136.45 

106.35 

Ilil 

mmm 

'll 

R-side  Comm.  Panel 

93.63 

98.47 

40.04 

81.04 

66.83 

92.79 

D-side  Map 

224.37 

101.74 

165.13 

111.11 

194.75 

109.03 

-!|li  M  -;..l 

■  ■  ©  ;  ;-5{ 

y. *■+#■;:  •  _r:J 

R-side  Map 

124.38 

129.53 

114.70 

117.72 

119.54 

121.85 

r.  S  -  ^ 

Keyboard  Area 

233.62 

66.51 

260.95 

32.75 

247.29 

53.41 

ji--  O  : 

mmw  J 

CRD 

300.04 

126.48 

263.67 

135.00 

281.86 

130.00 

ATWIT 

306.42 

64.75 

346.31 

272.43 

326.36 

195.83 

?s 

Flight  Strip  Bay  1 

198.38 

129.16 

176.22 

160.10 

187.30 

143.53 

Flight  Strip  Bay  2 

313.42 

48.09 

289.38 

46.56 

301.40 

48.14 

D-side  Comm.  Panel 

145.42 

152.60 

119.24 

117.72 

132.33 

136.20 

w.’. 

R-side  Comm.  Panel 

93.35 

98.61 

50.36 

73.58 

71.86 

89.20 

D-side  Map 

206.48 

106.03 

182.73 

111.90 

194.61 

109.08 

d 

R-side  Map 

165.04 

150.52 

173.76 

283.11 

169.40 

225.57 

(  § 

Keyboard  Area 

252.76 

66.76 

264.56 

48.07 

258.66 

58.17 

$  i  . 

CRD 

333.81 

172.63 

291.39 

148.22 

312.60 

161.45 

W^St^kri 

r  © 

f.r-W  ;  v 

ATWIT 

308.30 

146.51 

329.43 

181.19 

318.87 

164.24 

j^il 

Flight  Strip  Bay  1 

189.70 

108.85 

170.13 

125.64 

179.91 

117.34 

|rx:,v'.;;_;-:y  ,  .... 

jKu!  -  «1  *  i  . "1.  u  j* 

Flight  Strip  Bay  2 

324.53 

49.92 

297.40 

41.95 

310.96 

47.85 
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Table  W-12.  Saccade  Number:  Means  and  Standard  Deviations. 


Number 

Window 

'oad  1 

|  hHH 

Blf$|  'V 

Means  | 

SDs 

Means 

SDs 

Means  j 

SDs 

D-side  Comm.  Panel 

1.37 

1.92 

2.37 

3.30 

1.87 

2.71 

R-side  Comm.  Panel 

1.04 

1.72 

0.70 

1.11 

0.87 

1.44 

llBSlifcSillh 

D-side  Map 

5.76 

5.81 

5.85 

4.58 

5.81 

5.15 

£ 

R-side  Map 

5.33 

7.82 

4.94 

7.70 

5.14 

7.64 

wm&iipK**  *>  ,  >> ; 

Keyboard  Area 

178.26 

128.45 

283.54 

180.82 

230.90 

163.29 

feSlsig 

CRD 

7.98 

11.38 

13.16 

14.73 

10.57 

13.21 

.■■Si: 

ATWIT 

11.70 

9.16 

14.70 

9.48 

13.20 

9.29 

Flight  Strip  Bay  1 

22.90 

32.87 

27.28 

45.88 

25.09 

39.32 

Flight  Strip  Bay  2 

539.73 

432.07 

438.29 

346.43 

489.01 

388.66 

^§11 

D-side  Comm.  Panel 

2.12 

2.96" 

2.68 

3.72 

2.40 

3.32 

pjMBj 

R-side  Comm.  Panel 

0.73 

1.33 

0.13 

0.24 

0.43 

0.99 

III  § 

D-side  Map 

21.14 

39.35 

7.48 

9.03 

14.31 

28.93 

(?:"#■<  s  ‘  "V'T  ">) 

§ 

1  *2  >  i 

R-side  Map 

9.58 

16.43 

5.76 

9.28 

7.67 

13.27 

£* 

iMbrsH  I 

|  ;  S 

Keyboard  Area 

166.33 

122.45 

220.16 

174.56 

193.24 

150.82 

CRD 

10.67 

7.83 

9.10 

9.22 

9.88 

8.45 

ATWIT 

13.07 

8.31 

16.26 

7.92 

14.67 

8.15 

Flight  Strip  Bay  1 

3.84 

4.93 

9.66 

21.98 

6.75 

15.95 

;  a^I:-  ••■! 

Flight  Strip  Bay  2 

199.80 

119.86 

169.98 

94.06 

184.89 

107.06 

1WM 

D-side  Comm.  Panel 

6.56 

12.05 

8.81 

18.01 

7.68 

15.12 

../I 

R-side  Comm.  Panel 

0.38 

0.43 

0.23 

0.52 

0.30 

0.48 

'  JiSll 

D-side  Map 

14.73 

16.21 

7.45 

8.98 

11.09 

13.41 

^||§ 

R-side  Map 

7.57 

20.31 

1.46 

2.28 

4.51 

14.55 

Keyboard  Area 

93.16 

72.41 

188.95 

134.34 

141.06 

116.78 

CRD 

6.91 

8.39 

5.48 

5.11 

6.19 

6.87 

HMl 

ATWIT 

15.08 

6.92 

13.51 

8.09 

14.29 

7.45 

■ 

Flight  Strip  Bay  1 

4.78 

5.29 

4.27 

7.03 

4.53 

6.12 

ll^g 

Flight  Strip  Bay  2 

122.98 

93.68 

136.67 

87.87 

129.82 

89.61 

KIliiMS 

D-side  Comm.  Panel 

3.35 

7.46 

4.62 

10.97 

3.99 

9.35 

R-side  Comm.  Panel 

0.72 

1.28 

0.35 

0.75 

0.53 

1.06 

D-side  Map 

13.88 

25.09 

6.93 

7.69 

10.40 

18.78 

\  Sgj 

R-side  Map 

7.49 

15.50 

4.05 

7.19 

5.77 

12.14 

I^SsS  | 

Keyboard  Area 

145.92 

114.76 

230.88 

165.83 

188.40 

148.14 

CRD 

8.52 

9.27 

9.24 

10.71 

8.88 

9.97 

UHliij 

ATWIT 

13.28 

8.13 

14.82 

8.42 

14.05 

8.27 

j,j,,i 

Flight  Strip  Bay  1 

10.50 

20.98 

13.74 

30.66 

12.12 

26.18 

cjj  -  ♦  |  ■  .  ;  '  § 

Flight  Strip  Bay  2 

287.50 

316.95 

248.31 

249.42 

267.91 

284.37 
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TableW-13.  Saccade  Percent:  Means  and  Standard  Deviations. 


Percent 

Window 

nail 

QSSO 

Means 

SDs 

Means 

SDs 

•  •  j 

D-side  Comm.  Panel 

0.05 

0.07 

■i 

0.11 

•; .  1 

If-.. j 

R-side  Comm.  Panel 

0.04 

0.07 

■n 

0.03 

•"*■4  ’■ 

i . 

D-side  Map 

0.22 

0.21 

0.22 

0.17 

m  • 

■  C  >  •  1 

r- .  «w  ■■■■■■! 

■  '  r-  1 

R-side  Map 

0.20 

0.33 

0.17 

0.26 

Keyboard  Area 

6.63 

4.73 

10.08 

5.74 

!  1  4 

CRD 

0.29 

0.37 

0.49 

0.55 

<f‘,j 

ATWIT 

0.45 

0.37 

0.54 

0.34 

Flight  Strip  Bay  1 

0.85 

1.23 

1.00 

1.80 

i 

Flight  Strip  Bay  2 

20.16 

15.49 

15.99 

11.99 

D-side  Comm.  Panel 
R-side  Comm.  Panel 
D-side  Map 
R-side  Map 
Keyboard  Area 
CRD 
ATWIT 

Flight  Strip  Bay  1 
Flight  Strip  Bay  2 


D-side  Comm.  Panel 
R-side  Comm.  Panel 
D-side  Map 
R-side  Map 
Keyboard  Area 
CRD 
ATWIT 

Flight  Strip  Bay  1 
Flight  Strip  Bay  2 


D-side  Comm.  Panel 
R-side  Comm.  Panel 
D-side  Map 
R-side  Map 
Keyboard  Area 
CRD 
ATWIT 

Flight  Strip  Bay  1 
Flight  Strip  Bay  2 


6.40 

0.41 

0.52 

0.15 

7.89 


0.24 

0.02 

0.55 

0.26 

3.52 

0.25 

0.60 

0.18 

4.86 


0.12 

0.03 

0.52 

0.28 

5.52 

0.31 

0.52 

0.39 

10.97 


1.10 
L05 
.37 
i.67 
4.74 
0.29 
0.35 
0.18 
4.84 


0.45 

0.02 

0.54 

0.65 

2.57 

0.25 

0.31 

0.19 

3.52 


0.28 

0.05 

0.87 

0.56 

4.30 

0.31 

0.34 

0.78 

11.52 


Means 


0.06 

0.03 

0.22 

0.19 

8.35 

0.39 

0.49 

0.92 

18.08 


0.09 

0.02 

0.54 

0.29 

7.03 

0.37 

0.56 

0.25 

7.18 


0.26 

0.01 

0.41 

0.15 

5.04 

0.22 

0.54 

0.18 

4.91 


0.14 

0.02 

0.39 

0.21 

6.81 

0.32 

0.53 

0.45 

10.06 


SDs 


0.09 

0.05 

0.19 

0.29 

5.46 

0.47 

0.35 

1.52 

13.79 


0.12 

0.04 

1.02 

0.52 

5.05 

0.31 

0.32 

0.58 

4.31 


0.50 

0.02 

0.45 

0.46 

3.80 

0.21 

0.31 

0.24 

3.38 


.31 
.04 
.66 
.44 
4.96 
0.35 
0.32 
1.00 
10.26 
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Table  W-14.  Saccade  Duration:  Means  and  Standard  Deviations. 


Duration 

Window 

Means 

SDs 

1  fetU 

Means 

’  SDs 

Means 

SDs 

i- 

D-side  Comm.  Panel 

707.25 

1319.33 

911.18 

1272.79 

809.21 

1279.39 

jwpg 

R-side  Comm.  Panel 

634.75 

894.46 

344.52 

535.89 

489.64 

740.15 

JptSi 

D-side  Map 

2703.05 

2529.92 

2689.57 

2209.82 

2696.31 

2336.66 

R-side  Map 

2680.03 

2911.01 

2520.49 

3152.75 

2600.26 

2986.05 

Keyboard  Area 

2818.91 

955.97 

2703.07 

631.98 

2760.99 

799.33 

CRD 

1505.60 

1682.51 

1635.63 

869.56 

1570.61 

1319.09 

ATWIT 

1233.51 

666.79 

1175.88 

534.71 

1204.70 

595.26 

BSH 

Flight  Strip  Bay  1 

1670.76 

1573.31 

1501.31 

871.76 

1586.04 

1254.14 

Flight  Strip  Bay  2 

2853.83 

576.93 

2916.60 

952.17 

2885.21 

775.09 

mmm 

D-side  Comm.  Panel 

1216.61 

1713.92 

718.49 

725.34 

967.55 

1319.09 

R-side  Comm.  Panel 

589.89 

1184.04 

85.32 

152.62 

337.60 

869.10 

pm  || 

D-side  Map 

3125.15 

1628.48 

2906.64 

2537.37 

3015.90 

2100.19 

jjil  I 

R-side  Map 

2626.53 

2453.71 

3228.24 

3831.18 

2927.39 

3179.45 

1  1  1 

Keyboard  Area 

3059.28 

667.58 

3114.74 

718.45 

3087.01 

682.79 

CRD 

1604.61 

927.95 

1637.97 

1340.51 

1621.29 

1134.22 

ATWIT 

1468.61 

747.89 

1246.68 

442.38 

1357.64 

614.86 

flgg 

Flight  Strip  Bay  1 

738.13 

769.84 

1010.95 

864.71 

874.54 

817.18 

kUH 

Flight  Strip  Bay  2 

3053.89 

940.83 

2847.68 

607.04 

2950.78 

785.86 

ifem 

D-side  Comm.  Panel 

798.51 

885.04 

957.43 

1351.06 

877.97 

1126.40 

R-side  Comm.  Panel 

255.43 

318.57 

137.74 

309.65 

196.59 

314.76 

w  . 

D-side  Map 

2831.45 

2198.90 

1943.80 

1744.94 

2387.62 

2004.05 

:  1  ■ 

R-side  Map 

2471.99 

3163.32 

1569.41 

2183.80 

2020.70 

2712.88 

Keyboard  Area 

2627.37 

1057.79 

2978.97 

752.20 

2803.17 

920.37 

CRD 

1593.64 

1146.25 

2100.72 

1181.98 

1847.18 

1173.93 

§|l§fo 

ATWIT 

1419.48 

553.03 

1406.18 

848.49 

1412.83 

704.55 

ijailB 

Flight  Strip  Bay  1 

2919.02 

5923.20 

818.28 

948.61 

1868.65 

4307.03 

MttK 

Flight  Strip  Bay  2 

3194.61 

1381.53 

3587.12 

1922.13 

3390.86 

1658.61 

— 

D-side  Comm.  Panel 

907.46 

1339.12 

862.36 

1130.68 

884.91 

1232.96 

R-side  Comm.  Panel 

493.36 

874.30 

189.19 

377.46 

341.27 

687.05 

•  1*  1  ;» 

D-side  Map 

2886.55 

2112.84 

2513.33 

2181.44 

2699.94 

2144.30 

•  fl  ■ 

R-side  Map 

2592.85 

2797.77 

2439.38 

3138.55 

2516.12 

2958.36 

Keyboard  Area 

2835.19 

907.13 

2932.26 

709.04 

2883.72 

811.30 

CRD 

1567.95 

1264.76 

1791.44 

1144.35 

1679.70 

1204.94 

ATWIT 

1373.87 

654.59 

1276.25 

626.85 

1325.06 

639.37 

Flight  Strip  Bay  1 

1775.97 

3604.37 

1110.18 

923.47 

1443.08 

2638.42 

KlPM: 

Flight  Strip  Bay  2 

3034.11 

1008.87 

3117.13 

1303.72 

3075.62 

1160.26 
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.  Table  W-15.  DST  Display  Number:  Means  and  Standard  Deviations. 


Number 

Window 

■H9 

MB 

mu 

liusp 

HiH 

Means 

SDs 

Means 

SDs 

Means 

SDs 

,  ..  V... :  -  .  ^  -  ■  ;  •  £ 

Clock 

0.82 

1.76 

1.09 

1.95 

0.96 

1.83 

*■4 

,  V"-? 

Response  Display 

0.35 

0.75 

0.46 

0.83 

0.41 

0.78 

♦«;•••••• 

r 

Aircraft  List 

105.42 

226.64 

140.56 

251.44 

122.99 

236.15 

'  :  J3  '  ,  : 

Graphic  Plans  Display 

33.58 

72.19 

44.77 

80.09 

39.17 

75.22 

■  *K"'~ 

w- ! 

.  • 

• 

Trial  Plans 

0.47 

1.00 

0.62 

1.11 

0.54 

1.04 

"■  4’t \2 

Plans  Display 

3.40 

7.31 

4.54 

8.11 

3.97 

7.62 

-::i;  •• 

Other  Displays 

208.31 

61.70 

239.48 

149.62 

223.89 

113.69 

Clock 

3.07 

5.21 

2.40 

2.64 

2.74 

4.08 

e*  ‘ 

Response  Display 

1.97 

3.43 

0.59 

0.90 

1.28 

2.56 

55- 

•  c  :  1 

Aircraft  List 

804.23 

339.45 

765.68 

345.06 

784.96 

337.27 

«  -  ;-i! 
1,1 

Graphic  Plans  Display 

34.20 

71.90 

46.71 

79.05 

40.45 

74.60 

^mOVV^I 

•Mr.  :•  *:  ■  -r.- 

Trial  Plans 

1.22 

2.32 

2.75 

5.54 

1.98 

4.25 

r 

Plans  Display 

24.71 

48.29 

32.72 

67.10 

28.72 

57.65 

Hv  - .  "■! 

Other  Displays 

113.00 

70.98 

178.98 

123.86 

145.99 

104.81 

^ii  ■  I 

Clock 

8.47 

20.11 

4.38 

10.46 

6.42 

15.90 

1  -  *n~- 

e*  ■  ■"••••  ••■.-! 

Response  Display 

0.90 

1.82 

0.72 

1.15 

0.81 

1.50 

.  • .' .  ^#-L;rTi£^ 

Aircraft  List 

829.43 

314.49 

756.98 

251.82 

793.21 

282.66 

;  I  M 

Graphic  Plans  Display 

482.02 

484.57 

438.14 

415.54 

460.08 

444.60 

wm 

Trial  Plans 

2.00 

2.77 

1.78 

2.32 

1.89 

2.52 

?-  *  • 

Plans  Display 

22.91 

18.11 

15.46 

16.28 

19.19 

17.36 

Other  Displays 

153.16 

70.24 

197.00 

279.81 

175.08 

201.91 

Clock 

4.12 

12.21 

2.63 

6.34 

3.37 

9.71 

if1  T": 

Response  Display 

1.07 

2.33 

0.59 

0.96 

0.83 

1.79 

’•5'  §>  -;ii 

Aircraft  List 

579.70 

446.88 

554.41 

407.29 

567.05 

425.48 

4-  -  2  £  $ 

SsSiSjlrfl^ 

Graphic  Plans  Display 

183.27 

351.89 

176.54 

306.75 

179.90 

328.37 

« f  §  ■ 

Trial  Plans 

1.23 

2.21 

1.72 

3.56 

1.47 

2.96 

Plans  Display 

17.01 

31.00 

17.57 

40.99 

17.29 

36.15 

lllllisl 

Other  Displays 

158.16 

77.18 

205.15 

194.13 

181.65 

148.83 
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Table  W- 16.  DST  Display  Percent:  Means  and  Standard  Deviations. 


WKMM 

1  ITffllti 

oml 

jBiiiiiiigai 

Percent 

w  mao  w 

Means  | 

SDs 

Means  | 

SDs 

Means 

SDs 

Clock 

24.69 

53.08 

32.92 

58.89 

28.81 

55.31 

Response  Display 

24.67 

53.05 

32.90 

58.85 

28.79 

55.27 

Aircraft  List 

28.36 

60.96 

37.81 

67.63 

33.08 

63.52 

gPpn 

Graphic  Plans  Display 

25.79 

55.44 

34.39 

61.51 

30.09 

57.77 

HMgjB 

Trial  Plans 

24.68 

53.06 

32.91 

58.86 

28.79 

55.28 

< _ i| 

Plans  Display 

24.78 

53.28 

33.04 

59.11 

28.91 

55.52 

ppiw 

Other  Displays 

6.99 

1.99 

8.01 

5.43 

7.50 

4.06 

IBB  |||i|| 

Clock 

24.77 

53.05 

32.97 

58.86 

28.87 

55.28 

8  r|nj 

Response  Display 

24.73 

53.02 

32.90 

58.85 

28.82 

55.26 

LliilSI 

Aircraft  List 

52.59 

50.00 

59.41 

56.37 

56.00 

52.53 

Mm  * 

Graphic  Plans  Display 

25.81 

55.43 

34.45 

61.47 

30.13 

57.75 

j  2  |  | 

Trial  Plans 

24.70 

53.05 

32.99 

58.81 

28.85 

55.25 

|  <  || 

Plans  Display 

25.51 

52.95 

34.07 

58.55 

29.79 

55.08 

111 

Other  Displays 

3.93 

2.44 

6.06 

4.01 

5.00 

3.44 

jp  g||g 

Clock 

33.19 

58.74 

24.81 

53.03 

29.00 

55.21 

-  :f0 

Response  Display 

32.92 

58.84 

24.69 

53.04 

28.80 

55.26 

j  ' 

Aircraft  List 

62.86 

53.85 

50.80 

50.76 

56.83 

51.84 

Graphic  Plans  Display 

49.01 

54.56 

39.39 

50.40 

44.20 

51.90 

Trial  Plans 

32.95 

58.83 

24.73 

53.04 

28.84 

55.26 

Plans  Display 

33.70 

58.72 

25.20 

53.08 

29.45 

55.23 

j|  jj  |S 

Other  Displays 

5.49 

2.63 

6.79 

9.84 

6.14 

7.11 

ggflgPgg 

Clock 

27.55 

53.99 

30.23 

55.90 

28.89 

54.68 

isspii 

Response  Display 

27.44 

54.00 

30.16 

55.89 

28.80 

54.68 

IQpBg 

Aircraft  List 

47.93 

55.88 

49.34 

58.11 

48.64 

56.71 

Graphic  Plans  Display 

33.54 

55.08 

36.08 

56.83 

34.81 

55.68 

Jfe; 

Trial  Plans 

27.45 

54.01 

30.21 

55.88 

28.83 

54.68 

^0)fe 

Plans  Display 

28.00 

54.02 

30.77 

55.89 

29.38 

54.69 

— J'' 

Other  Displays 

5.47 

2.64 

6.95 

6.79 

6.21 

5.18 
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Table  W-17.  DST  Display  Duration:  Means  and  Standard  Deviations. 


Duration 

Window 

ins 

I 

|^Xoa3Coliapse<£ 

Means 

SDs 

Means 

SDs 

Means 

SDs 

Clock 

286.43 

615.81 

381.91 

683.18 

334.17 

641.63 

fc  J  rv : 

■  '  £*•  •  ■ 

Response  Display 

253.77 

545.59 

338.36 

605.28 

296.06 

568.46 

-  a.  :•  ••  , 

Aircraft  List 

315.48 

678.27 

420.65 

752.48 

368.07 

706.71 

Graphic  Plans  Display 

282.65 

607.69 

376.87 

674.17 

329.76 

633.16 

■  a::-.  .  * 

Trial  Plans 

277.72 

597.09 

370.30 

662.41 

324.01 

622.12 

'•  '  y  '• . 

Plans  Display 

291.31 

626.29 

388.41 

694.81 

339.86 

652.55 

a- - ,  ,v;:;K 

Other  Displays 

365.94 

57.80 

370.39 

75.14 

368.17 

65.98 

'”™,  T1 

l:  •  ■  n 

*  ■'  -  i 

Clock 

510.29 

613.75 

634.51 

645.35 

572.40 

622.71 

.rspvv;} 

Response  Display 

337.97 

526.23 

360.23 

598.52 

349.10 

554.49 

..-ft:  : 

a 

.  •  •— i  ■ 

t'-r.-.-i  ,  S-'l;:: r| 

Aircraft  List 

771.91 

459.49 

818.90 

524.88 

795.41 

485.84 

"  .-Ci. 

'  : S' 

Graphic  Plans  Display 

342.77 

600.24 

452.33 

652.19 

397.55 

619.06 

Trial  Plans 

465.51 

734.06 

444.49 

639.47 

455.00 

677.28 

< 

r 

Plans  Display 

504.73 

607.29 

554.31 

641.61 

529.52 

615.05 

SSi . J 

Other  Displays 

351.81 

113.02 

389.66 

104.15 

370.74 

108.62 

Clock 

658.93 

667.75 

539.19 

693.86 

599.06 

672.61 

ml  \ 

S  '  1 

Response  Display 

353.83 

599.18 

311.06 

543.36 

332.45 

563.07 

|J:::  O  '  .  J 

Aircraft  List 

856.05 

497.92 

767.68 

458.24 

811.87 

472.85 

t.:"M  U 

:r-  v- 

Graphic  Plans  Display 

769.66 

450.08 

664.85 

441.04 

717.25 

441.56 

-  .  CV:r4'.^ 

;  +-*  V  -  ■ 

sl'r-  51- 

Trial  Plans 

619.85 

702.71 

551.87 

580.51 

585.86 

634.97 

Plans  Display 

835.91 

522.02 

594.31 

526.17 

715.11 

529.98 

^  1 

Other  Displays 

429.17 

95.60 

443.03 

123.55 

436.10 

108.89 

in 

Clock 

485.22 

638.35 

518.54 

668.28 

501.88 

650.25 

(  c  -j 

Response  Display 

315.19 

547.66 

336.55 

570.87 

325.87 

556.53 

,  'Tt;' ©  . j  u  • :  ~  J 

Aircraft  List 

647.81 

592.40 

669.08 

606.31 

658.45 

596.33 

liptl 

Graphic  Plans  Display 

465.03 

587.79 

498.02 

598.39 

481.52 

590.21 

3  5  ! 

Trial  Plans 

454.36 

680.69 

455.56 

619.49 

454.96 

647.37 

Plans  Display 

543.98 

617.35 

512.34 

618.01 

528.16 

614.63 

BH 

Other  Displays 

382.31 

95.99 

401.03 

105.35 

391.67 

100.69 

Table  W-18.  Fixations:  MANOVA  Results. 


Wilks 

Pillai's 

Hotelling 

Roy's 

F 

Load 

.506 

0.494 

0.975 

0.975 

1.625 

6 

■Q 

WKEM 

Automation 

.129 

0.871 

6.727 

6.727 

2.242 

H 

K] 

.227 

Load  X  Automation 

.404 

0.596 

1.475 

1.475 

0.492 

12 

4 

BBSS 
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TableW-19.  Fixation  Number:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

52522 

Load 

205977.011 

msm 

■a 

■i*W 

Automation 

101956.010 

mmu 

Load  X  Automation 

3480.456 

mmmm 

M 

■in 

Table  W-20.  Fixation  Mean  Duration:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df2 

122223 

Load 

■HSIQ 

mm 

.069 

Automation 

HE 

Til 

.004 

Load  X  Automation 

0.002 

0.004 

30 

.606 

Table  W-21.  Fixation  SD  Duration:  ANOVA  Results. 


Load 

■E 

Automation 

0.073 

■EME 

■ 

Til 

Load  X  Automation 

0.000 

0.056 

2 

30 

.945 

Table  W-22.  Fixation  Mean  Area:  ANOVA  Results. 


F 

Load 

apoi 

Automation 

30 

.191 

B 

1 

Table  W-23.  Fixation  SD  Area:  ANOVA  Results. 


msm 

F 

£0223 

Load 

1.266 

1.341 

0.944 

■E 

15 

.347 

Automation 

1.195 

2.394 

2 

30 

.108 

Load  X  Automation 

2.897 

1.583 

1.830 

2 

30 

.178 

Table  W-24.  Visual  Efficiency:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

0.017 

0.003 

■H 

.022 

Automation 

0.011 

0.003 

m 

H 

Load  X  Automation 

0.004 

0.003 

m 

RH 

Table  W-25.  Saccades:  MANOVA  Results. 


asgfMI 

p-level 

Load 

■U3GI 

■HfetRl 

3 

.155 

Automation 

mm 

■HB 

E 

.087 

Load  X  Automation 

.360 

0.640 

mum 

4 

.783 

Table  W-26.  Saccades  Number:  ANOVA  Results. 


Load _ 

Automation 
Load  X  Automation 


IMS  Effect 


|Ms  Error 


450781.205 

38616.885 

26212.230 


69993.299 

131725.035 

101018.952 


F 

6.440 

0.293 

0.259 


df  lldf  2\p-level 


1  15 

2  30 
2  30 


.023 

.748 

.773 
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Table  W-27.  Saccades  Mean  Duration:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

0.004 

0.001 

4.493 

1 

15 

.051 

Automation 

0.003 

0.001 

2.360 

2 

30 

.112 

Load  X  Automation 

0.001 

0.001 

1.464 

2 

30 

.247 

Table  W-28.  Saccades  SD  Duration:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

0.009 

0.016 

0.523 

1 

15 

.481 

Automation 

0.014 

0.016 

0.880 

2 

30 

.425 

Load  X  Automation 

0.017 

0.014 

1.237 

2 

30 

.305 

Table  W-29.  Saccades  Mean  Distance:  ANOVA  Results. 


1 

Load 

wlmmm 

■SEES 

mu  it: 

mm 

u 

wfcm 

1.158 

0.217 

5.340 

mm 

■SEES 

Table  W-30.  Saccades  SD  Distance:  ANOVA  Results. 


F 

Load 

725.268 

821.772 

0.883 

1 

15 

.362 

Automation 

1244.119 

1077.287 

1.155 

2 

30 

.329 

Load  X  Automation 

595.648 

1224.598 

0.486 

2 

30 

.620 

Table  W-31.  Eye  Motion  Workload:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

1.317 

0.463 

2.843 

1 

15 

.112 

Automation 

Bug! 

2 

30 

mi Li 

Load  X  Automation 

flEUE] 

2 

a—ciT? 

Table  W-32.  Blinks:  MANOVA  Results. 


Hotelling 

Roy's 

mm 

df  1 

df2 

1SS9S 1 

Load 

Si 

IlfcMl 

msgi 

■E 

■0 

.546 

Automation 

msi 

ItffIB 

ME 

.630 

Load  X  Automation 

.767 

0.233 

0.304 

0.304 

0.182 

10 

6 

.991 

Table  W-33.  Blinks  Number:  ANOVA  Results. 


MS  Effect 


Ms  Error 


Load 

Automation 
Load  X  Automation 


4169.333 

13392.969 

10749.561 


12275.937 

20692.823 

24963.446 


df  1 


0.340 


0.6471 


0.431 


df  2 

1  15 

2  30 

2  30 


p-level 

369 

.531 

.654 


Table  W-34.  Blinks  Mean  Duration:  ANOVA  Results. 


Ms  Error 

F 

Load 

0.232 

m 

■E 

.896 

Automation 

0.005 

.963 

Load  X  Automation 

0.049 

0.473 

0.104 

2 

30 

.902 

W-16 


Table  W-35.  Blinks  SD  Duration:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df2 

p-level 

Load 

0.002 

0.268 

0.009 

1 

15 

.926 

Automation 

0.038 

0.165 

0.230 

2 

30 

.796 

Load  X  Automation 

0.050 

0.476 

0.105 

2 

30 

.900 

Table  W-36.  Blinks  Mean  Distance:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

1.120 

19.292 

0.058 

1 

15 

.813 

Automation 

17.008 

8.198 

2.075 

2 

30 

.143 

Load  X  Automation 

1.308 

11.728 

0.111 

2 

30 

.895 

Table  W-37.  Blinks  SD  Distance:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

357674.224 

112335.839 

3.184 

1 

15 

.095 

Automation 

105250.398 

128555.372 

0.819 

2 

30 

.451 

Load  X  Automation 

86300.386 

146171.334 

0.590 

2 

30 

.560 

Table  W-38.  Scene  Based  Fixations:  MANOVA  Results. 


Wilks 

Pillai's 

Hotelling 

Roy's 

F 

df  1 

df  2 

p-level 

Load 

.780 

0.220 

0.281 

0.281 

1.220 

3 

13 

.342 

.140 

6.168 

6.168 

10 

.001 

Display 

SH 

13 

.006 

Load  X  Automation 

■tfcfcVJ 

BEi 

6 

10 

.573 

Load  X  Display 

.896 

0.116 

0.116 

3 

13 

.686 

Automation  X  Display 

.074 

0.926 

12.484 

12.484 

20.807 

6 

10 

.000 

Load  X  Auto  X  Display  ! 

.775 

0.225 

0.291 

0.291 

0.485 

6 

10 

.806 

Table  W-39.  Scene  Based  Fixations  Number:  ANOVA  Results. 


MS  Effect 

ETil 

my 

Load 

52501.932 

ME 

Automation 

999761.109 

H3 

iwti 

Radar  Display 

1328190.318 

42815.702 

31.021 

2 

30 

.000 

DST  Display 

4570467.771 

■ail 

Display 

1 

ME 

No  Automation 

9457444.133 

127484.516 

74.185 

1 

■E 

Limited  Automation 

303615.281 

KrtiKtaHl 

bees 

1 

ME 

mme 

Full  Automation 

2451728.320 

1 

ME 

Load  x  Automation 

5377.126 

2 

30 

.912 

Load  x  Display 

11875.521 

213391.265 

0.056 

1 

15 

.817 

Automation  x  Display 

10797555.068 

215660.268 

50.067 

2 

30 

.000 

Load  x  Auto  x  Display 

115364.255 

268737.666 

0.429 

2 

30 

.655 

W-17 


Table  W-40.  Scene  Based  Fixations  Percent:  ANOVA  Results. 


.utomation 


Radar  Display 


DST  Display 

Display 

MS  EffectMS  Erro 


2  0. 
5  29. 


I  0.138!  0.005|26.252 


No  Automation 

1.045  0.015  70.339 

Limited  Automation 
Full  Automation 

T _ 1  A.-i _ -  - 

0.031  0.025  1.261 

1.113  0.008143.952 

Load  x  Display 


utomation  x  Display 


_ isp.  „ 


Table  W-41 .  Scene  Based  Fixations  Mean  Duration:  ANOVA  Results. 


Load 

Automation 


Radar  Display 


DST  Display 


Display 


No  Automation 


Limited  Automation 


Full  Automation 


Load  x  Automation 


Load  x  Display 
Automation  x  Display 


MS  Effect  MS  Error 
0.010  0.007 

0.087  0.008 


df 1 Idf 2 


0.007  1.540 

0.008  1  1.338' 


p-level 

234 

.000 


Table  W-42.  Scene  Based  Saccades:  MANOVA  Results. 


Wilks Pillai’s  Hotelling  Roy's 

F 

.258  0.742  2.871  2.871 

12.441 

Display 


Load  X  Automation 


Load  X  Display 


Automation  X  Display 
Load  X  Auto  X  Displa 


.143  0.857 
.288  0.712 
.957  0.0431 


075 

390 


025.869 
2.472  2.472  4.120' 
0.044  0.044  0.193" 
12.39912.39920.665" 
1.562  1.562  2.603' 


8 


Table  W-43.  Scene  Based  Saccades  Number:  ANOV A  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-level 

Load 

42271.444 

26148.328 

1.617 

1 

m 

Automation 

1331755.641 

61728.259 

21.574 

2 

H3 

•g 

Radar  Display 

985900.911 

37370.679 

26.382 

2 

18! 

.000 

DST  Display 

4600148.583 

93418.945 

49.242 

m 

METT 

mm 

Display 

227012.521 

372171.310 

0.610 

m 

m 

No  Automation 

6108823.195 

98975.962 

61.720 

Ml 

m 

Limited  Automation 

5113.133 

177532.999 

0.029 

m 

m 

Full  Automation 

3839913.281 

109427.681 

kmgeh 

K 

ME 

Load  x  Automation 

3382.203 

72021.773 

m 

Load  x  Display 

17787.000 

205936.367 

i 

m 

Automation  x  Display 

9840343.349 

199850.988 

2 

Load  x  Auto  x  Display 

103618.734 

231039.184 

0.448 

2 

30 

.643 

Table  W-44.  Scene  Based  Saccades  Percent:  ANOYA  Results. 


F 

BMI 

Qggj 

Load 

m 

fe TA 

Automation 

0.175 

30 

Radar  Display 

0.146 

30 

DST  Display 

0.633 

m 

30 

Display 

0.032 

l 

m 

mkse 

No  Automation 

0.014 

60.837 

1 

15 

gg 

Limited  Automation 

■HEBE 

0.026 

0.014 

1 

15 

ME 

mmm 

BE 

Load  x  Display 

m 

ME 

mm 

Automation  x  Display 

1 

57.957 

2 

■381 

Load  x  Auto  x  Display 

0.019 

0.030 

0.638 

2 

.535 

Table  W-45.  Scene  Based  Saccades  Mean  Distance:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

Load 

2.606 

0.110 

23.643 

1 

15 

No  Automation 

2.189 

0.135 

16.215 

1 

15 

Limited  Automation 

0.054 

0.000 

1 

15 

Full  Automation 

0.044 

5.593 

1 

15 

1  jg 

Automation 

iHim 

0.167 

3.510 

2 

30 

Low  Load 

m 

High  Load 

0.063 

0.057 

1.116 

2 

.341 

Radar  Display 

m 

DST  Display 

4.254 

Display 

16.755 

0.212 

15 

No  Automation 

0.256 

22.878 

i 

15 

Limited  Automation 

1.166 

0.037 

31.828 

1 

15 

.000 

Full  Automation 

2.285 

0.040 

57.332 

1 

15 

.000 

Load  x  Automation 

1.134 

0.178 

6.371 

2 

30 

.005 

Load  x  Display 

0.015 

0.120 

0.127 

1 

15 

.726 

Automation  x  Display 

0.939 

0.227 

4.139 

2 

30 

.026 

Load  x  Auto  x  Display 

1.139 

0.214 

5.317 

2 

30 

.011 

W-19 


Table  W-46.  Scene  Based  Saccades  Mean  Distance-  Simple  Effects  of  Load  x  Automation  x 

Display:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df2 

/Mevel 

Load  X  Auto  X  Display 

1.139 

0.214 

5.317 

2 

30 

.011 

Low  Load 

Automation 

1.593 

0.231 

6.889 

2 

30 

.003 

Display 

7.879 

0.217 

36.338 

i 

15 

.000 

Automation  X  Display 

2.059 

0.295 

6.968 

2 

30 

.003 

High  Load 

Automation 

0.127 

0.114 

1.116 

2 

30 

.341 

Display 

8.891 

0.115 

77.268 

1 

15 

.000 

Automation  x  Display 

0.020 

0.146 

0.137 

2 

30 

.873 

No  Automation 

Load 

4.378 

0.270 

16.215 

1 

15 

.001 

Display 

11.732 

0.513 

22.878 

1 

15 

.000 

Load  x  Display 

1.316 

0.274 

4.813 

1 

15 

.044 

Limited  Automation 

Load 

0.000 

0.107 

0.000 

1 

15 

.998 

Display 

2.331 

0.073 

31.828 

1 

15 

.000 

Load  x  Display 

0.705 

0.181 

3.901 

1 

15 

.067 

Full  Automation 

Load 

0.497 

0.089 

5.593 

1 

15 

.032 

Display 

4.570 

0.080 

57.332 

1 

15 

.000 

Load  x  Display 

0.272 

0.094 

2.889 

i 

15 

.110 

Radar  Display 

Load 

2506319.448 

11842.404 

211.639 

1 

15 

.000 

Automation 

2503368.923 

1 1837.607 

211.476 

2 

30 

.000 

Load  x  Automation 

2502110.968 

11843.250 

211.269 

2 

30 

.000 

DST  Display 

Load 

1.111 

0.162 

6.844 

1 

15 

.019 

Automation 

1.435 

0.337 

4.254 

2 

30 

.024 

Load  x  Automation 

2.253 

0.272 

8.272  2 

30 

.001. 

Table  W-47.  Object  Based:  MANOVA  Results. 


Wilks 

Pillai's 

Hotelling 

Roy's 

F 

df  1 

df  2 

p-level 

Load 

.607 

0.393 

0.646 

0.646 

4.523 

2 

14 

.031 

Automation 

.216 

_ 

0.784 

3.620 

3.620 

10.861 

4 

12 

.001 

Information 

.024 

0.976 

40.780 

40.780 

285.458 

2 

14 

.000 

Load  X  Automation 

.964 

0.036 

0.038 

0.038 

0.114 

4 

12 

.975 

Load  X  Information 

.651 

0.349 

0.536 

0.536 

3.755 

2 

14 

.049 

Automation  X  Information 

.213 

0.787 

3.705 

3.705 

11.116 

4 

12 

.001 

Load  X  Auto  X  Information 

.687 

0.313 

0.455 

0.455 

1.366 

4 

12 

.303 
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Table  W-48.  Object  Based  Number:  ANOVA  Results. 


MS  Effect  I  MS  Error 


Load 

171945.060 

Automation 

1026609.536 

FDB 

515681.078 

Position  Symbol 

89344.786 

Idf  1 1  df  2  Ip- 


Automation  x  Information 
Load  x  Auto  x  Information 


183442.193  11778.452 
2595.622  12241.572" 


0.006 


15.574 

0.212" 


Table  W-49.  Object  Based  Duration:  ANOVA  Results. 


MS  Effect  MS  Error  F 

^oad 

0.005  0.003  1.914 

Position  Symbol 


Automation _ 

Information 


Low  Load 
High  Load 


Load  x  Automation _ 

Load  x  Information 


Automation  x  Information 
Load  x  Auto  x  Information 


Table  W-50.  Structure  Conditional  Index:  MANOVA  Results. 


Table  W-52.  Range  Conditional  Index:  ANOVA  Results. 


Table  W-53.  Box  Conditional  Index:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df2 

p~levei 

Load 

5.335 

■ 

KE 

.036 

Automation 

8.403 

m 

KB 

Load  X  Automation 

0.001 

0.032 

0.044 

m 

30 

.957 

Table  W-54.  Ring  Conditional  Index:  ANOVA  Results. 


ISMaMWI 

df  1 

df  2 

p-levei 

Load 

1 

15 

.664 

Automation 

1.778 

2 

30 

.186 

Load  X  Automation 

0.002 

0.443 

2 

30 

.646 

W-22 


APPENDIX  X 

Controller-to-Controller  Coordination  Communications  Taxonomy 


Table  X-l .  Controller-to-Controller  Coordination  Communications  Taxonomy  (C4T): 

Coordination-Communication  Topic. 


ATC 

Coordination- 

Communication 

Topic 

Definitions  and  Examples 

Approval 

Communications  about  intersector  control/approval  requests  (“Get  me 
control  for  descent  on  that  aircraft.”  “APREQ  N1234  climbing  to  FL330.”) 

Handoff 

Communications  relating  to  the  transfer  of  radar  identification  of  a 
particular  aircraft  (“Handoff  N1234.”  “Did  you  handoff  N1234.”) 

Point  Out 

Communications  relating  to  the  transfer  of  radar  identification  of  a 
particular  aircraft  when  radio  communications  will  be  retained  (“Point  out 
N1234  to  22.”) 

Traffic 

Communications  about  a  traffic  situation  involving  a  specific  aircraft. 
Includes  conflict,  spacing,  other  protected  air  space  or  terrain  and  the 
resolution  of  that  situation.  (“Are  you  watching  that  aircraft.”) 

Altitude 

Communications  about  altitude  not  in  relation  to  traffic  (“N1234  is 
requesting  flight  level  220.”) 

Route 

Communications  regarding  headings  and/or  amendments  to  route,  not  in 
relation  to  traffic  situations  (“N1234  is  on  a  330  heading.”  “Next  sector, 

27,  wants  N 1234  over  WEVER.”) 

Speed 

Communications  about  speed  not  in  relation  to  traffic  situations  (“These 
three  aircraft  are  slowed  to  250  knots.”) 

Weather 

Communications  about  weather  display  or  weather  updates  (Often 
communicated  nonverbally  by  passing  written  information.  “Sector  22  says 
continuous  moderate  turbulence  above  FL290.”  ) 

Frequency 

Communications  about  an  aircraft’s  radio  communications  transfer  or 
frequency  assignment  (“Have  you  switched  N1234  yet?”  “Tell  them  to 
switch  to  N1234.”) 

Flow  Messages 

Communications  about  traffic  flow  restrictions  not  referring  to  a  specific 
aircraft  (“The  next  sector  is  requesting  25  miles  in  trail.”)  (due  to  radar 
outage) 

Flight  Strips 

Communications  about  flight  progress  strips  (“Where  is  that  strip?”)  Often 
communicated  nonverbally 

Equipment 

Communications  about  any  ATC  hardware  (The  radar  is  out  of  service.”) 

Aircraft  ID 
Identification  of 
Aircraft 

Communications  involving  identifying  a  specific  aircraft  (Who  was  that 
who  called?”  “That  was  N1234  who  called.”) 

X-l 


Table  X-2.  Controller-To-Controller  Coordination  Communications  Taxonomy  (C4T): 

Coordination-Communication  Format. 


ATC 

Coordination- 

Communication 

Format 

Definitions 

Question 

A  direct  inquiry  about  the  state  or  status  of  sector  events. 

Answer 

A  response  to  a  direct  or  implied  question 

Statement 

Providing  information,  without  being  asked,  about  the  state  or  status  of 
sector  events. 

Command 

A  direct  order  to  perform  a  specific  act 

X-2 


Table  X-3.  Controller-To-Controller  Coordination  Communications  Taxonomy  (C4T): 
Coordination-Communication  Expression. 


ATC 

Coordination- 

Communication 

Expression 

Definitions 

Verbal 

Use  of  voice  only  communication. 

Nonverbal 

Use  of  only  body  movement  communication. 

Mixed 

Communication  that  contains  both  a  verbal  and  non  verbal  component. 

Electronic 

Not  used.  Communication  that  is  electronically  transferred. 

X-3 


Table  X-4.  Frequency  for  Topic  of  Communication:  Means  and  Standard  Deviations  (number). 


Low  Load 


Means  SDs 


Total  Communications 
Approval 


Automation  1 


Automation  2 


Handoff 

3.33 

1.53 

Traffic 

24.40 

6.43 

Altitude 

8.80 

6.53 

Route  of  Flight 

7.40 

3.36 

Speed 

3.20 

2.68 

Frequency 

3.40 

1.94 

Flight  Strips 

4.33 

4.16 

Equipment 

6.00 

8.66 

Total  Communications 

62.60 

27.63 

Approval 

1.00 

0.00 

Handoff 

5.00 

5.66 

Traffic 

26.20 

21.71 

Altitude 

8.80 

9.26 

Route  of  Flight 

5.80 

4.55 

Speed 

4.80 

5.93 

Frequency 

2.67 

0.58 

Flight  Strips 

N/A 

N/A 

Equipment 

5.00 

3.87 

Automation  ! 
I  Collapsed  ■  < 


Total  Communications 

Approval 

Handoff 

Traffic 

Altitude 

Route  of  Flight 

Speed 

Frequency 

Flight  Strips 

Equipment 


Total  Communications 

Approval 

Handoff 

Traffic 

Altitude 

Route  of  Flight 

Speed 

Frequency 

Flight  Strips 

Equipment 


High  Load 


Means  SDs 


Load  Collapsed 


Means 


.10 

26.3 

50 

0.7 

60 

1.5: 

40 

8.8: 

40 

10.3 

.80 

6.9' 

38 

2.21 

56 

2.6< 

67.60 

19.49 

2.33 

1.53 

3.40 

3.36 

30.80 

17.61 

11.20 

7.84 

9.70 

8.46 

4.22 

4.29 

2.14 

0.69 

N/A 

N/A 

4.33 

3.12 

60.80 

23.71 

2.50 

2.38 

3.60 

3.78 

32.50 

13.58 

11.90 

5.63 

!  7.20 

7.25 

»  4.00 

4.47 

i  3.00 

1.07 

N/A 

N/A 

5.57 

2.76 

i  64.83 

22.73 

2.22 

1.72 

3.20 

2.86 

30.23 

13.50 

12.50 

7.99 

8.23 

7.40 

3.87 

3.60 

2.96 

1.78 

2.33 

2.46 

5.10 

4.02 

X-4 


Table  X-5.  Frequency  for  Expression  of  Communication:  Means  and  Standard  Deviations 

(number). 


Expression  of 
Communication 

Question 

Answer 

Statement _ 

Question 

Answer 

Statement _ 

Question 

Answer 

Statement 

Question 

Answer 

Statement 


Table  X-6.  Frequency  for  Communication  Expression:  Means  and  Standard  Deviations 

(number). 


Communication 

Expression 

*  ■*  .  ^ 

i 

1 

Load  Collapsed 

Low  Load 

High  Load 

Means 

SDs 

Means 

SDs 

Means 

SDs 

Verbal 

46.40 

18.30 

52.80 

24.10 

20.45 

AUto"iam'n  ' 

Mixed  Verbal/Nonverbal 

6.80 

7.05 

13.80 

. 7 

Verbal 

53.60 

55.20 

8.76 

54.40 

17.11 

Automation  2 

Mixed  Verbal/Nonverbal 

6.60 

14.60 

10.60 

6.74 

Verbal 

51.60 

23.55 

17.90 

19.88 

Automation  | 

Mixed  Verbal/Nonverbal 

7.80 

4.66 

8.40 

8.14 

6.26 

Verbal 

50.53 

20.73 

54.80 

16.79 

52.67 

18.66 

Mixed  Verbal/Nonverbal 

7.07 

4.92 

12.27 

7.76 

9.67 

6.91 

Table  X-7.  Total  Number  of  Communications:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

197.633 

188.717 

1.047 

1 

4 

Automation 

127.633 

326.175 

0.391 

2 

8 

Load  X  Automation 

68.433 

406.892 

0.168 

2 

8 

Table  X-8.  Traffic:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

440.833 

39.167 

11.255 

1 

4 

.028 

Automation 

67.433 

119.892 

0.562 

2 

8 

.591 

Load  X  Automation 

6.433 

182.892 

0.035 

2 

8 

.966 

X-5 


Table  X-9.  Altitude:  ANOVA  Results. 


MS  Effect 


Ms  Error 


Load 

Automation 
Load  X  Automation 


187.500 

28.300 

93.100 


17.583 

35.300 

61.683 


df  1 


10.664 


0.802 


1.509 


df2 


2 

2 


p-level 


4 

8 

8 


.031 

.482 

.278 


Table  X-10.  Route  of  Flight:  ANOVA  Results. 


F 

Load 

20.833 

12.417 

1.678 

m 

Hi 

.265 

Automation 

8 

Load  X  Automation 

8 

■Hid 

Table  X-ll.  Speed:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df  2 

p-level 

Load 

0.333 

EH 

1 

1 

■B5H 

Automation 

mm 

Load  X  Automation 

30.333 

1.132 

.469 

Table  X-12.  Frequency:  ANOVA  Results. 


Load 


Automation 


Load  X  Automation 


MS  Effect  Ms  Error  F  dfl  df2  p-level 


0.333 


10.083 


1.083 


5.333 


4.083 


10.083 


0.063 


2.469 


0.107 


.844| 

.288 


.903 


Table  X-13.  Question:  ANOVA  Results. 


Load 


Automation 


Load  X  Automation 


MS  Effect 


!Ms  Error 


13.333 


46.433 


10.033 


10.333 


17.183 


17.533 


1.290 


2.702 


0.572 


dfl 


Idf 2  ]p -level 


.319 


.127 


.586 


Table  X-14.  Answer:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

ESS 

Load 

1 

■ 

.642 

Automation 

2 

8 

Load  X  Automation 

15.633 

32.342 

0.483 

!  2 

8 

■era 

TableX-15.  Statement:  ANOVA  Results. 


BT1H 

Load 

H 

H 

.061 

Automation 

8 

Load  X  Automation 

2 

8 

Ha 

X-6 


TableX-16.  Verbal:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p4evel 

Load 

136.533 

176.783" 

0.772 

1 

4 

.429" 

Automation 

70.933 

132.142: 

0.537 

2 

i 

8 

.604 

Load  X  Automation 

14.933 

275.308 

0.054 

2 

8 

.948 

Table  X-17.  Mixed  Verbal/Nonverbal:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

202.800 

62.217 

3.260 

1 

4 

.145 

Automation 

18.633 

21.217 

0.878 

8 

.452 

Load  X  Automation 

40.300 

22.967 

1.755 

2 

8 

.233 

X-7 


APPENDIX  Y 


Push-to-Talk 

Table  Y-l .  Push-to-Talk:  Means  and  Standard  Deviations  for  Number  and  Duration  of  Landline 

Communications  (number;  seconds). 


L  Comraunicatloi 

Table  LL 

Low  Load 

High  Load 

Load  Coll. 

Means 

SDs 

Means 

SDs 

Means 

SDs 

|So7: 

N 

0.26 

0.29 

0.43 

0.19 

0.35 

0.25 

D 

1.28 

0.95 

1.81 

0.40 

1.56 

0.73 

• f  p 

N 

0.19 

0.18 

0.60 

0.37 

0.40 

0.35 

D 

1.20 

0.92 

1.80 

0.21 

1.50 

0.71 

gipgjlj 

N 

0.44 

0.22 

0.68 

0.48 

0.56 

0.38 

D 

2.45 

0.71 

1.77 

0.59 

2.11 

0.72 

N 

0.30 

0.25 

0.57 

0.36 

0.44 

0.34 

IB 

D 

1.68 

1.01 

1.79 

0.42 

1.74 

0.76 

Table  Y-2.  Push-to-Talk:  Means  and  Standard  Deviations  for  Number  and  Duration  of  R-side 

Communications  (number;  seconds). 


it  v 

HfflRRj 

1— 

n 

Table  R 

Low  Load 

High  Load 

Load  Coll. 

Means 

SDs 

Means 

SDs 

Means 

SDs 

N 

3.66 

0.57 

5.38 

0.76 

l.n 

i'njki5:? 

D 

3.18 

0.24 

3.09 

0.18 

0.21 

KgMg| 

N 

3.80 

0.36 

5.40 

0.62 

4.60 

0.97 

refill 

D 

3.18 

0.31 

2.87 

0.27 

3.03 

0.32 

l- 

N 

3.81 

0.48 

5.54 

0.46 

1.00 

D 

3.10 

0.22 

3.01 

0.19 

HI 

3.05 

0.20 

Auto 

N 

3.76 

0.46 

5.45 

0.60 

4.62 

1.00 

■Hf 

D 

3.15 

0.25 

3.00 

0.22 

3.07 

0.24 

Y-l 


Table  Y-3.  Mean  Number/Duration  PTT  Landline  Communications:  MANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

F 

df  1 

df  2 

p-level 

Load 

.293 

0.707 

2.408 

2.408 

7.225 

2 

6 

.025 

Automation 

.114 

0.886 

7.800 

7.800 

7.800 

4 

4 

.036 

Load  X  Automation 

.126 

0.874 

6.959 

6.959 

6.959 

4 

4 

.043 

Table  Y-4.  Mean  Number  PTT  Landline  Communications:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df2 

p-level 

Load 

0.752 

0.046 

16.180 

1 

7 

.005 

Automation 

0.183 

0.051 

3.606 

2 

14 

.055 

Load  X  Automation 

0.045 

0.098 

0.457 

2 

14 

.642 

Table  Y-5.  Mean  Duration  PTT  Landline  Communications:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df  2 

p-level 

Load 

0.130 

0.088 

1.472 

1 

7 

.264 

No  Automation 

0.869 

0.331 

2.627 

1 

7 

.149 

Limited  Automation 

1.103 

0.338 

3.265 

1 

7 

.114 

Full  Automation 

1.842 

0.167 

11.031 

1 

7 

.013 

Automation 

1.680 

0.571 

2.942 

2 

14 

.086 

Low  Task  Load 

3.519 

0.791 

4.447 

2 

14 

.032 

High  Task  Load 

0.002 

0.153 

0.015 

2 

14 

.985 

Load  x  Automation 

1.842 

0.374 

4.931 

2 

14 

.024 

Table  Y-6.  Mean  Number/Duration  PTT  Radar-side  (R)  Communications:  MANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

F 

dfl 

df  2 

p-level 

Load 

.023 

0.977 

43.402 

43.402 

13.206 

2 

6 

.000 

Automation 

.650 

0.350 

0.538 

0.538 

0.538 

4 

4 

.719 

Load  X  Automation 

.484 

0.516 

1.067 

1.067 

1.067 

4 

4 

.476 

Table  Y-7.  Mean  Number  PTT  Radar-side  (R)  Communications:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df  2 

p-level 

Load 

29.900 

0.202 

148.068 

1 

7 

.000 

Automation 

0.040 

0.256 

0.156 

2 

14 

.857 

Load  X  Automation 

0.109 

0.320 

0.341 

2 

14 

.717 

Table  Y-8.  Mean  Duration  PTT  Radar-Side  (R)  Communications:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df  2 

p-level 

Load 

0.248 

0.021 

11.735 

1 

7 

.011 

Automation 

0.041 

0.024 

1.680 

2 

14 

.222 

Load  X  Automation 

0.047 

0.015 

3.202 

2 

14 

.072 

Y-2 


APPENDIX  Z 
DST  Trust  Results 


Table  Z-l .  DST  Trust  Means  and  Standard  Deviations. 


Question 


1 5)  Predictable  Behavior 

1 6)  Predicting  Outcome  of  Sep.  Strategies 

1 8)  Predicting  Future  Conflicts 

19)  Technical  Knowledge 

1 5)  Predictable  Behavior 

1 6)  Predicting  Outcome  of  Sep.  Strategies 

1 8)  Predicting  Future  Conflicts 

19)  Technical  Knowledge _ 

1 5)  Predictable  Behavior 

16)  Predicting  Outcome  of  Sep.  Strategies 

1 8)  Predicting  Future  Conflicts 

19)  Technical  Knowledge _ 


i— B 

ssi 

snbijt 

J|i  .  . 

i  BBB5 

MS 

i  mi  ■ 

pg|pB|f 

Load/Pos. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Collapsed 

EE9 

EE9 

mEsa 

IdlM 

ESjSSSI 

5.25 

1.52 

4.85 

1.98 

5.05 

1.75 

5.61 

2.32 

5.82 

1.84 

5.72 

2.06 

5.43 

1.94 

5.34 

1.95 

5.38 

1.93 

5.14 

1.19 

4.83 

2.13 

4.99 

1.71 

5.02 

2.41 

5.19 

1.76 

5.11 

2.08 

5.08 

1.87 

5.01 

1.93 

5.05 

1.89 

5.77 

1.54 

5.41 

2.20 

5.59 

1.88 

5.77 

2.31 

5.95 

1.97 

5.86 

2.11 

5.77 

1.93 

5.68 

5.72 

1.99 

1.51 

5.87 

1.63 

5.84 

1.55 

6.00 

2.39 

6.00 

1.67 

6.00 

2.03 

5.90 

1.97 

5.93 

1.62 

5.92 

1.79 

5.48 

1.86 

4.72 

2.14 

5.10 

2.01 

6.51 

2.05 

4.80 

2.47 

5.65 

2.40 

5.99 

1.99 

4.76 

2.27 

5.37 

2.21 

5.84 

1.86 

5.03 

1.60 

5.44 

1.76 

6.70 

2.23 

5.17 

2.58 

5.94 

2.49 

6.27 

5.10 

2.11 

5.69 

2.15 

5.85 

2.05 

5.41 

1.91 

5.63 

1.96 

6.72 

2.06 

5.61 

2.47 

6.16 

2.31 

6.28 

US 

5.51 

2.17 

5.90 

2.14 

5.93 

1.65 

5.62 

1.58 

5.77 

1.60 

6.87 

1.79 

5.87 

2.09 

6.37 

1.98 

1.76 

5.75 

1.83 

6.07 

1.81 

5.36 

1.36 

4.89 

KIF1 

mm 

■Eg 

By 

1.79 

5.58 

ran 

5.59 

1.61 

5.11 

1.78 

5.35 

1.71 

5.46 

1.28 

5.08 

1.52 

5.27 

1.41 

5.70 

1.99 

5.25 

1.77 

5.48 

1.89 

5.58 

1.67 

5.17 

1.64 

5.37 

1.67 

5.82 

1.45 

5.56 

1.66 

5.69 

1.55 

6.11 

1.80 

5.80 

1.79 

5.96 

1.79 

5.97 

1.63 

5.68 

1.72 

5.82 

1.68 

5.91 

1.27 

5.82 

1.29 

5.87 

1.27 

6.29 

1.74 

5.95 

1.52 

6.12 

6.10 

1.53 

5.89 

1.40 

5.99 

1.47 

Table  Z-2.  DST  Trust  Items:  MANOV A  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

mm 

ESQ 

Position 

.837 

0.163 

0.195 

0.195 

IjQIQ 

4 

HEM 

Load 

.650 

0.350 

0.537 

0.537 

4 

ME 

Automation 

.376 

0.624 

1.656 

1.656 

■E 

WE 

Position  X  Load 

.733 

0.267 

0.365 

0.365 

HE 

WE 

Si 

Position  X  Automation 

.783 

0.217 

0.277 

0.277 

■E 

.569 

■Sal 

0.756 

4 

WE 

mmm 

.777 

0.286 

urns 

4 

WE 

Table  Z-3.  The  Predictability  of  the  DST’s  Behavior:  ANOVA  Results. 


MS  Error 

F 

BUI 

Position 

■HM1I 

6.821 

1.748 

m 

WE 

HESS 

Load 

1.951 

7.212 

1 

WE 

TilR 

Limited  Automation 

Mm® 

HKSfifi 

1 

WE 

Full  Automation 

mmm 

HESS 

1 

SHE 

■Kim 

Automation 

1 

WE 

Low  Load 

2.508 

1.025 

1 

15 

;  139 

High  Load 

1 

WE 

.045 

Position  x  Load 

SHKMS 

mi 

WE 

.785 

Position  x  Automation 

1 

15 

.828 

Load  x  Automation 

mwm 

1 

15 

.007 

Position  x  Load  x  Automation 

1.824 

2.676 

1 

15 

.123 

Z-l 


Table  Z-4.  The  Predictability  of  the  DST’s  Behavior:  ANOVA  Results  (N=6). 


MS  Effect 

MS  Error 

F 

df  1 

df2 

p- level 

Position 

3.201 

9.017 

0.355 

1 

5 

.577 

Load 

14.083 

2.779 

5.068 

5 

.074 

Automation 

0.941 

2.691 

0.350 

5 

.580 

Position  X  Load 

0.464 

4.282 

0.108 

5 

.755 

Position  X  Automation 

3.000 

1.947 

1.541 

5 

.270 

Load  X  Automation 

7.302 

1.958 

3.728 

5 

.111 

Position  X  Load  X  Automation 

8.333 

0.813 

1.254 

5 

.024 

R  side 

Load 

6.554 

0.493 

13.297 

1 

5 

.015 

Automation 

2.317 

1.405 

1.649 

2 

10 

.240 

Load  x  Automation 

1.647 

0.602 

2.735 

2 

10 

.113 

D  side 

Load 

3.144 

4.214 

0.746 

1 

5 

.427 

Automation 

2.020 

3.421 

0.590 

2 

10 

.572 

Load  x  Automation 

8.595 

1.960 

4.385 

2 

10 

.043 

Table  Z-5.  How  Well  the  DST  Predicted  the  Outcome  of  the  Separation  Strategies: 

ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/7-Ievel 

Position 

3.082 

5.910 

0.522 

1 

15 

.481 

Load 

12.259 

3.691 

3.321 

1 

15 

.088 

Automation 

13.109 

1.904 

6.883 

1 

15 

.019 

Position  X  Load 

.102 

2.819 

0.036 

1 

15 

.852 

Position  X  Automation 

1.137 

2.801 

0.406 

1 

15 

.534 

Load  X  Automation 

9.659 

2.616 

3.693 

1 

15 

.074 

Position  X  Load  X  Automation 

2.869 

1.225 

2.342 

1 

15 

.147 

Table  Z-6.  Capability  of  the  Tool  to  Predict  Future  Conflicts:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df2 

p-level 

Position 

5.261 

8.245 

0.638 

1 

15 

.437 

Load 

5.934 

1.171 

5.068 

1 

15 

.040 

Automation 

0.950 

1.594 

0.596 

1 

15 

.452 

Position  X  Load 

0.036 

1.101 

0.033 

1 

15 

.859 

Position  X  Automation 

0.556 

1.500 

0.371 

1 

15 

.552 

Load  X  Automation 

3.832 

1.724 

2.223 

1 

15 

.157 

Position  X  Load  X  Automation 

2.989 

2.843 

1.051 

1 

15 

.321 

Table  Z-7.  The  Technical  Knowledge  Incorporated  in  the  Tool:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df2 

p-level 

Position 

4.619 

5.105 

0.905 

1 

15 

.357 

Load 

3.121 

1.277 

2.445 

1 

15 

.139 

Automation 

0.794 

0.798 

0.995 

1 

15 

.334 

Position  X  Load 

1.132 

1.410 

0.802 

1 

15 

.385 

Position  X  Automation 

1.529 

1.213 

1.260 

1 

15 

.279 

Load  X  Automation 

3.777 

1.202 

3.142 

1 

15 

.097 

Position  X  Load  X  Automation 

0.772 

1.333 

0.579 

1 

15 

.458 

Z-2 


APPENDIX  AA 


DRAT  Results 


Table  AA-1.  Interval  Altitude:  Means  and  Standard  Deviations. 


Interval 


Interval  Collapsed 


i  4-' 

ipni 

jgvimi 

Means  I 

SDs 

Means 

SDs 

Means  | 

SDs 

4.68 

1.53 

4.77 

1.70 

4.73 

1.59 

3.24 

1.72 

4.34 

1.71 

3.79 

1.78 

2.93 

1.37 

4.46 

1.38 

3.70 

1.56 

3.31 

1.24 

4.34 

2.03 

3.82 

1.74 

3.74 

1.56 

3.52 

1.44 

3.63 

1.48 

2.99 

1.35 

3.90 

1.73 

3.45 

1.59 

3.37 

1.74 

3.71 

1.38 

3.54 

1.55 

3.12 

1.13 

4.15 

1.52 

3.63 

1.42 

2.12 

1.65 

3.15 

1.40 

2.63 

1.59 

2.62 

1.13 

3.34 

1.61 

2.98 

1.41 

3.21 

1.56 

3.97 

1.64 

3.59 

1.64 

3.99 

1.81 

4.14 

4.07 

1.48 

3.24 

1.21 

4.08 

1.25 

3.66 

1.28 

2.93 

1.02 

4.77 

1.69 

3.85 

1.66 

3.36 

0.92 

3.64 

1.60 

3.50 

1.29 

2.74 

0.96 

2.83 

0.88 

2.78 

0.91 

3.18 

1.14 

3.64 

1.56 

3.41 

1.36 

3.18 

1.58 

4.08  . 

1.30 

3.63 

1.49 

3.24 

0.81 

4.21 

1.48 

3.72 

1.27 

2.11 

1.10 

2.89 

1.12 

2.50 

1.16 

2.36 

1.11 

3.33 

1.10 

2.85 

1.19 

3.03 

1.27 

3.76 

1.42 

3.40 

1.39 

4.46 

1.33 

4.30 

0.92 

4.38 

1.13 

2.65 

1.26 

4.49 

1.40 

3.57 

1.61 

2.96 

1.17 

4.43 

1.95 

3.69 

1.75 

3.15 

1.13 

4.05 

1.32 

3.60 

1.29 

3.09 

2.01 

3.18 

1.01 

3.13 

1.56 

3.11 

0.92 

3.49 

1.24 

3.30 

1.09 

2.74 

1.02 

3.86 

1.24 

3.30 

1.25 

2.61 

0.91 

4.11 

1.34 

3.36 

1.36 

2.36 

1.33 

2.86 

1.76 

2.61 

1.55 

2.49 

1.05 

3.30 

1.42 

2.90 

1.30 

2.96 

1.34 

3.81 

1.46 

3.39 

1.46 

4.38 

1.56 

4.41 

1.29 

4.39 

1.42 

3.04 

1.41 

4.30 

1.45 

3.67 

1.56 

2.94 

1.17 

4.55 

1.66 

3.75 

1.64 

3.27 

1.08 

4.01 

1.67 

3.64 

1.45 

3.19 

1.59 

3.18 

1.15 

3.18 

1.38 

3.10 

1.13 

3.68 

1.50 

3.39 

1.35 

3.10 

1.47 

3.89 

1.29 

3.49 

1.43 

2.99 

0.98 

4.16 

1.42 

3.57 

1.35 

2.20 

1.36 

2.97 

1.43 

2.58 

1.44 

2.49 

1.08 

3.32 

1.36 

2.91 

1.29 

3.07 

1.40l 

3.85 

1.51 

3.46 

Table  AA-2.  Interval  Heading:  Means  and  Standard  Deviations. 


AA-2 


Table  AA-3.  Interval  Speed:  Means  and  Standard  Deviations. 


AA-3 


Table  AA-4.  Interval  Handoffs  Accepted:  Means  and  Standard  Deviations. 


Means  | 
XXXXX 
3.20 
3.01 

4.58 
4.08 
3.95 
3.39 
2.64 
3.26 

3.58 

3.58 


XXXXX 


)s 

XXXXX 

2.05 

1.60 

1.67 

1.06 

1.57 

2.00 

1.05 

2.06 

1.83 

1.66 


XXXXX  I 


Means  | 
XXXXX 
4.71 
4.71 

4.52 
5.40 
5.27 
4.08 

5.52 
4.83 
5.7-1 


4.89 


XXXXX 


SDs 

XXXXX 

1.61 

1.61 

2.12 

1.69 

2.04 

0.68 

1.12 

1.59 

1.42 


1.57 


XXXXXI 


Means 

XXXXX 

3.96 

3.86 

4.55 

4.74 
4.61 

3.74 
4.08 
4.05 
4.64 


.23 


XXXXX 


Interval  Collapsed 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 


Interval  Collapsed 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10  _ 

Interval  Collapsed 


2.90 

1.61 

4.35 

1.53 

3.63 

.  3.22 

1.40 

4.04 

1.03 

3.63 

4.15 

1.09 

4.54 

1.78 

4.35 

4.34 

0.78 

5.16 

1.60 

4.75 

3.65 

1.27 

4.54 

1.08 

4.10 

3.03 

1.06 

4.54 

0.87 

3.78 

3.28 

1.15 

4.97 

1.09 

4.13 

3.34 

1.32 

4.47 

1.49 

3.91 

2.84 

1.25 

4.91 

1.23 

3.88 

3.49 

1.25 

4.56 

1.28 

4.03 

XXXXX 

4.15 
4.53 
4.53 
5.28 

5.15 

4.15 
5.28 
5.72 
4.09 


XXXXX 

1.41 


XXXXX 

3.34 


3.49 

1.45 

I  4.70 

1.55 

4.10 

XXXXX  XXXXX 

XXXXX  XXXXX  XXXXX 

2.88 

1.80 

4.40 

1.50 

3.64 

3.11 

1.62 

4.43 

1.32 

3.77 

4.27 

1.37 

4.53 

1.83 

4.40 

4.27 

1.14 

5.28 

1.70 

4.78 

3.88 

1.22 

4.99 

1.76 

4.43 

3.19 

1.56 

4.26 

1.03 

3.73 

2.90 

1.15 

5.26 

1.08 

4.08 

3.25 

1.63 

5.01 

1.68 

4.13 

3.34 

1.38 

4.90 

1.45 

4.12 

3.52 

1.46 

4.72 

1.47 

4.12 

SDs 

XXXXX 

1.97 
1.80 
1.88 
1.54 
1.91 
1.51 
1.81 

1.98 
1.94 


XXXXX 

1.71 

1.28 

1.46 

1.31 

1.24 

1.23 

1.40 

1.50 

1.61 


1.37 


XXXXX 

1.78 

1.76 

1.50 

1.74 

1.60 

1.55 

1.69 

2.09 

1.10 


1.61 


XXXXX 

1.82 

1.62 

1.61 

1.52 

1.61 

1.42 

1.62 

1.87 

1.62 

1.58 


We  omitted  the  first  interval  of  data  due  to  automatic  system  handoff  acceptance. 
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Table  AA-5.  Interval  Handoffs  Initiated:  Means  and  Standard  Deviations. 


Interval 


Means 


XXXXX 


SDs 


XXXXX 


Means 


XXXXX 


SDs 


XXXXX 


Means 


XXXXX 


-2.57 

1.94 

-3.20 

1.84 

-2.89 

-4.13 

1.86 

-4.83 

2.00 

-4.48 

-2.82 

1.43 

-3.83 

1.47 

-3.32 

-3.13 

1.37 

-4.33 

1.74 

-3.73 

-4.13 

2.03 

-5.95 

1.86 

-5.04 

-4.32 

1.78 

-4.20 

2.04 

-4.26 

-3.88 

1.96 

-6.08 

1.90 

-4.98 

-3.82 

1.64 

-4.45 

1.26 

-4.14 

-2.82 

1.28 

-4.33 

1.58 

-3.57 

-3.57 

1.70 

-4.52 

1.83 

-4.05 

5 

6 

7 

8 

9 

10 


Interval  Collapsed 


1 


5 

6 

7 

8 

9 

10 


Interval  Collapsed 


1 


XXXXX 

-3.08 

-3.39 

-3.64 

-3.95 

-3.52 

-3.89 

-3.33 

-3.64 

-4.27 


-3.68 


XXXXX 

-3.01 

-3.51 

-3.08 

-3.45 

-4.45 

-3.45 

-3.77 

-3.14 

-3.27 


-3.52 


XXXXX 

-2.89 

-3.68 

-3.18 

-3.51 

-4.03 

-3.89 

-3.66 

-3.53 

-3.45 


XXXXX 

2.02 

1.42 

1.41 

1.57 

1.60 

1.82 

1.36 

1.32 

2.77 


1.65 


XXXXX 

1.33 

1.94 

1.53 

1.79 

1.67 

1.32 

2.47 

1.64 

1.30 


1.63 


XXXXX 

1.77 


XXXXX 

-3.96 

-4.52 

-4.27 

-4.02 

-5.08 

-4.83 

-4.96 

-5.15 

-4.52 


-4.54 


XXXXX 

-3.14 

-4.27 

-4.08 

-4.33 

-5.33 

-5.02 

-6.08 

-4.89 

-4.52 


-4.57 


XXXXX 

-3.43 

-4.54 

-4.06 

-4.23 

-5.45 

-4.68 

-5.70 

-4.83 

-4.45 


-4.54 


XXXXX 

1.61 

2.36 

1.43 

1.10 

1.97 

2.63 

2.31 

1.88 

1.58 


1.83 


XXXXX 

1.68 

1.73 

1.95 

1.85 

1.65 

1.85 

3.54 

2.33 

1.41 


2.07 


XXXXX 

1.72 


XXXXX 

-3.52 

-3.96 

-3.96 

-3.99 

-4.30 

-4.36 

-4.14 

-4.39 

-4.39 


-4.11 


XXXXX 

-3.08 

-3.89 

-3.58 

-3.89 

-4.89 

-4.24 

-4.92 

-4.02 

-3.89 


-4.05 


XXXXX 

-3.16 

-4.11 

-3.62 

-3.87 

-4.74 

-4.29 

-4.68 

-4.18 

-3.95 


-4.07 


».«wri5E»j|f  Interval  Collapsed  I  -3.59 _ 1-66|  -4.54 _ 1.911  -4.07 _ 

We  omitted  the  first  interval  of  data  due  to  automatic  system  handoff  acceptance. 


SDs 


XXXXX 

1.89 

1.93 

1.52 

1.65 

2.13 

1.88 

2.20 

1.47 

1.61 


1.83 


XXXXX 

1.85 

2.00 

1.44 

1.33 

1.94 

2.28 

2.04 

1.77 

2.22 


1.79 


XXXXX 

1.49 

1.85 

1.80 

1.85 

1.69 

1.77 

3.22 

2.17 

1.48 


1.93 


XXXXX 

1.75 

1.92 
1.60 
1.61 

1.93 
1.97 
2.55 
1.81 
1.81 


1.851 
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SCENARIO  RESULTS 


Table  AA-6.  Altitude:  ANOVA  Results. 


MS  Effect 

Ms  Error 

P 

df  1 

df  2 

p-level 

Load 

HESSS 

HESSES 

■1 

7 

.030 

Automation 

HESSES 

1.017 

2 

14 

.387 

Load  X  Automation 

0.003 

0.002 

1.909 

2 

14 

.185 

Table  AA-7.  Heading:  ANOVA  Results. 


MS  Effect 

Ms  Error 

p 

p-level 

Load 

0.001 

■HEEZ* 

m 

m 

.276 

HESSES 

2 

m 

.592 

Load  X  Automation 

hseses 

0.660 

2 

14 

.532 

Table  AA-8.  Speed:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

SOI 

p-level 

Load 

0.004 

m 

■ 

mm 

Automation 

0.001 

0.001 

2 

14 

.583 

Load  X  Automation 

0.002 

0.002 

1.402 

2 

14 

.279 

Table  AA-9.  Handoffs  Accepted:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

0.001 

0.002 

0.610 

1 

m 

HEEE! 

Automation 

0.002 

0.001 

1.096 

2 

14 

HUS 

Load  X  Automation 

0.001 

0.004 

0.204 

2 

14 

.817 

Table  AA-10.  Handoffs  Initiated:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df2 

p-level 

Load 

0.016 

2.018 

1 

7 

.198 

Automation 

0.002 

0.003 

0.503 

2 

14 

.615 

Load  X  Automation 

0.005 

0.004 

1.297 

2 

14 

.304 
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INTERVAL  RESULTS 


Table  AA-11.  Altitude:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

p-level 

Load 

mmm 

36.365 

1 

Interval  1 

0.026 

1 

Interval  2 

■rasa 

1 

7 

Interval  3 

ESSU 

1 

7 

Interval  4 

2.172 

0.641 

3.391 

1 

7 

.108 

Interval  5 

0.001 

0.344 

0.002 

1 

7 

Interval  6 

0.224 

1 

7 

.044 

Interval  7 

2.496 

0.171 

14.589 

1 

7 

.007 

Interval  8 

5.428 

0.177 

3.645 

1 

7 

.001 

Interval  9 

2.366 

0.321 

7.361 

1 

7 

.030 

Interval  10 

2.766 

0.089 

31.026 

1 

7 

.001 

Automation 

2.114 

1.450 

1.458 

2 

14 

.266 

Interval 

11.719 

0.915 

12.812 

9 

63 

.000 

Low  Load 

2.563 

0.248 

1.345 

9 

63 

.000 

High  Load 

2.368 

0.311 

7.611 

9 

63 

.000 

Load  x  Automation 

0.152 

1.204 

0.126 

2 

14 

.883 

Load  x  Interval 

3.075 

0.762 

4.035 

9 

63 

.000 

Automation  x  Interval 

0.506 

0.989 

3.622 

18 

126 

.949 

Load  x  Automation  x  Interval 

0.404 

0.952 

4.145 

18 

126 

.980 

Table  AA-12.  Altitude:  Tukey  HSD  Post  Hoc  Test  Results. 


Interval  1 

Interval  2 

Interval  3 

Interval  4 

Interval  5 

Interval  6 

Interval  7 

Interval  8 

Interval  9 

Interval  10 

BESSKifl 

4.392 

3.673 

3.746 

3.642 

3.183 

■mi 

2.907 

Interval  1 

■HIE 

■EES 

Interval  2 

MIS 

■HUE 

■EES 

HBME 

mmm 

Interval  3 

■HES 

■hee 

■ese; 

Bflgggg 

Interval  4 

.374 

■HSi 

EEEE 

.013 

Interval  5 

■"Iprc- 

.918 

■ 

■ 

■ESI 

.313 

■ 

■K^E! 

■ 

Interval  8 

mm 

1.000 

.000 

.035 

Interval  9 

SB 

.000 

■ 

■ 

.816 

Interval  10 

.816 

TableAA-13.  Heading:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-level 

_______ 

0.335 

17.323 

m 1 

7 

.004 

Automation 

mmm 

0.660 

1.221 

EE 

.325 

Interval 

1.325 

9 

63 

.000 

Load  X  Automation 

0.443 

1.398 

0.317 

m 

14 

.734 

Load  X  Interval 

■bees 

0.418 

1.176 

m 

MSB 

.326 

Automation  X  Interval 

■Ei£E! 

0.287 

0.502 

m 

.953 

Load  X  Automation  X  Interval 

1.236 

18 

.243 
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Table  AA-14.  Heading:  Tukey  HSD  Post  Hoc  Test  Results. 


Interval  1 

Interval  2 

Interval  3 

Interval  4 

Interval  5 

Interval  6 

Interval  7 

Interval  8 

Interval  9 

Interval  10 

.436 

.624 

.561 

.739 

.436 

.610 

.714 

.818 

.975 

.704 

Interval  1 

.544 

.926 

.040 

1.000 

.649 

.081 

.003 

.000 

.108 

Interval  2 

.544 

.999 

.956 

1.000 

.991 

.493 

.996 

Interval  3 

.926 

.999 

.623 

.926 

HESS 

.792 

.140 

HEm 

.851 

Interval  4 

.040 

.956 

.623 

■EBl 

flUSsE 

HEEE 

1.000 

Interval  5 

1.000 

.544 

.926 

.040 

_ , 

.649 

■KBE 

.108 

Interval  6 

.649 

1.000 

1.000 

.913 

.976 

.393 

.005 

.988 

Interval  7 

.081 

.991 

.792 

1.000 

.081 

.976 

.976 

.128 

1.000 

Interval  8 

.003 

.493 

.140 

.996 

.003 

.393 

.976 

.770 

.956 

Interval  9 

.000 

.008 

.001 

.228 

.000 

.005 

.128 

.770 

.098 

Interval  10 

.108 

.996 

.851 

1.000 

.108 

.988 

1.000 

.956 

.098 

Table  AA-15.  Speed:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

p-  level 

Load 

9.766 

0.161 

6.616 

1 

7 

.000 

Automation 

0.153 

0.879 

0.174 

2 

14 

.842 

Interval 

2.666 

.354 

7.539 

9 

63 

.000 

Load  X  Automation 

0.077 

0.673 

0.114 

2 

14 

.893 

Load  X  Interval 

0.400 

0.254 

1.577 

9 

63 

.142 

Automation  X  Interval 

0.313 

0.343 

0.914 

18 

126 

.564 

Load  X  Automation  X  Interval 

0.210 

0.323 

0.648 

18 

126 

.855 

Table  AA-16.  Speed:  Tukey  HSD  Post  Hoc  Test  Results. 


Interval  1 

Interval  2 

Interval  3 

Interval  4  Interval  5 

Interval  6 

Interval  7 

Interval  8 

Interval  9 

Interval  10 

.234 

.373 

.467 

.686 

.571 

.932 

1.005 

.682 

.515 

.650 

Interval  1 

.000 

.000 

.016 

.396 

.034 

Interval  2 

.978 

.828 

.001 

.000 

.268 

.975 

.414 

Interval  3 

■Kl 

■BEE 

.997 

.010 

.002 

.752 

1.000 

.883 

Interval  4 

.585 

.227 

1.000 

.921 

1.000 

HI 

.108 

.022 

.996 

1.000 

1.000 

Interval  6 

.000 

.001 

.010 

.585 

.108 

1.000 

.562 

.033 

.393 

Interval  7 

.000 

.000 

.002 

.227 

.022 

1.000 

.213 

.006 

.122 

Interval  8 

.016 

.268 

.752 

1.000 

.996 

.562 

.213 

.931 

1.000 

Interval  9 

.396 

.975 

1.000 

.921 

1.000 

.033 

.006 

.931 

.981 

Interval  10 

.034 

.414 

.883 

1. 000 

1.000 

.393 

.122 

1.000 

.981 
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TableAA-17.  Handoffs  Accepted:  ANOVA  Results. 


Table  AA-1 8.  Handoffs  Accepted:  Tukey  HSD  Post  Hoc  Test  Results. 


Table  AA-20.  Handoffs  Initiated:  Tukey  HSD  Post  Hoc  Test  Results. 


Interval  2 

Interval  3 

Interval  4 

Interval  5 

Interval  6 

Interval  7 

Interval  8 

Interval  9 

Interval  10 

-3.161 

-4.108 

-3.619 

-3.870 

-4.745 

-4.286 

-4.682 

-4.182 

-3.953 

Interval  2d 

.005 

.585 

.085 

.000 

.001 

.000 

.002 

.035 

Interval  3 

.005 

.497 

.983 

.170 

.998 

.285 

1.000 

•  .999 

Interval  4 

.585 

.497 

.977 

.001 

.128 

.001 

.308 

.886 

Interval  5 

.085 

.983 

.977 

.013 

.701 

.027 

.919 

1.000 

Interval  6 

.000 

.170 

.001 

.013 

.585 

1.000 

.310 

.035 

Interval  7 

.001 

.998 

.128 

.701 

.585 

.755 

1.000 

.887 

Interval  8 

.000 

.285 

.001 

.027 

1.000 

.755 

.468 

.069 

Interval  9 

.002 

1.000 

.308 

.919 

.310 

1.000 

.468 

.987 

Interval  10 

.035 

.999 

.886 

1.000 

.035 

.887 

.069 

.987 

a  We  omitted  the  first  interval  of  data  due  to  automatic  system  handoff  acceptance. 


Table  AA-21.  Number  and  Duration  of  Conflicts:  Means  and  Standard  Deviations 

(number;  seconds). 


Conflicts 

s 

Load 

Collapsed 

Low  Load 

High  Load 

Means  SDs 

Means  |  SDs 

Means]  SDs 

f  ‘•;1.  rV^vv* 

rAupilfi 

Number 

Duration 

0.19  0.40 

4.75  11.52 

0.33  0.49 

13.53  31.25 

0.26  0.44 

9.00  23.28 

:  Auto  2 

Number 

Duration 

0.27  0.46 

16.40  49.42 

0.50  0.90 

11.00  20.95 

0.37  0.69 
14.00  38.84 

Auto  3 

Number 

Duration 

0.27  0.46 

21.27  69.55 

0.53  0.52 

8.47  9.52 

0.40  0.50 

14.87  49.21 

mi 

Number 

Duration 

0.24  0.43 

13.93  48.57 

0.45  0.63 

11.00  22.06 

0.34  0.54 

12.53  38.10 

Table  AA-22.  Number  of  Aircraft  Handled  Per  Minute:  Means  and  Standard  Deviations. 


■EsaaBesseansa 

Aircraft 

Handled 

Mllg 

Low  Load 

High  Lo 

m 

ad 

Load  Coll. 

Low  Load 

ggaggi 

Load/Pos. 

Collapsed 

EBE2I 

umi 

Ejgii 

155851 

EE9 

ES5SI 

EI53M  Number 

1.79  0.22 

2.57  0.29 

2.18 

0.47 

1.79  0.22 

2.57 

0.29 

2.18 

0.47 

1.79 

0.21 

2.57 

0.29 

2.18 

0.47 

Auto  2  '! 

Number 

1.80  0.23 

2.47  0.28 

2.13 

0.43 

1.80  0.23 

2.47 

0.28 

2.13 

0.43 

1.80 

0.23 

2.47 

0.28 

2.13 

0.42 

Number 

1.79  0.24 

2.49  0.29 

2.14 

0.44 

1.79  0.24 

2.49 

0.29 

2.14 

0.44 

1.79 

0.24 

2.49 

0.29 

2.14 

0.44 

Auifolll 

Number 

1.79  0.22 

2.51  0.29 

2.15 

0.44 

1.79  0.22 

2.51 

0.29 

2.15 

0.44 

1.79 

0.22 

2.51 

0.29 

2.15 

0.44 
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Table  AA-23.  Number  of  Aircraft  Handled  Per  Minute:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df  2 

/7-level 

Position 

0.059 

0.000 

1 

15 

1.000 

Load 

0.092 

270.952 

1 

15 

.000 

Automation 

muu 

mm 

30 

MuE 

Position  X  Load 

mm® 

15 

■Mi 

Position  X  Automation 

Load  X  Automation 

.336 

Position  X  Load  X  Automation 

0.000 

0.055 

0.000 

2 

30 

1.000 
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APPENDIX  BB 


Controller  Interactions  with  DSR 


Table  BB-1.  Information  Pickup:  Means  and  Standard  Deviations  (number). 


1 

Information  Pickup 

^  Low  Load 

High  Load 

~  '  Load  Coll. 

5 '  Low  Loa<i  ' 

isnni 

High  Load 

— 

gBE 

Load/Pos. 

Collapsed 

Essai 

E&3 

Means  ( 

SDs 

Means  \ 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  j 

SDs 

Means  | 

SDs 

Total  Number  of  Entries 

265.17 

42.55 

331.40 

46.39 

295.27 

54.43 

100.67 

61.60 

124.80 

63.77 

111.64 

60.68 

182.92 

99.64 

228.10  120.15 

203.45  109.53 

Flight  Plan  Readouts 

7.50 

6.38 

5.60 

3.97 

6.64 

5.26 

0.00 

0.00 

0.00 

0.00 

0.00 

3.75 

5.82 

2.80 

3.97 

3.32 

4.97 

J.  : 

J-rings 

2.67 

2.34 

2.40 

1.67 

2.55 

1.97 

0.67 

1.21 

0.40 

0.89 

0.55 

1.04 

1.67 

2.06 

1.40 

1.65 

4.97 

1.55 

Route  Displays 

2.33 

2.42 

0.60 

0.55 

1.55 

1.97 

1.17 

2.04 

0.40 

0.55 

0.82 

1.54 

1.75 

2.22 

0.50 

0.53 

1.18 

1.76 

L ... 

Total  Number  of  Entries 

279.50 

12.82 

342.67 

72.14 

317.40 

63.32 

24.50 

22.29 

92.50 

32.92 

65.30 

44.73 

152.00  137.34 

217.58  141.16 

191.35  139.90 

[ 

Flight  Plan  Readouts 

6.25 

4.35 

7.33 

7.65 

6.90 

6.26 

0.00 

0.00 

0.00 

0.00 

0.00 

3.13 

4.39 

3.67 

6.43 

3.45 

5.58 

J-rings 

1.75 

3.50 

2.67 

2.66 

2.30 

2.87 

0.25 

0.50 

0.67 

1.21 

0.50 

0.97 

1.00 

2.45 

1.67 

2.23 

1.40 

2.28 

■ 

Route  Displays 

4.00 

3.56 

1.50 

1.76 

2.50 

2.76 

1.25 

1.89 

1.17 

0.47 

1.20 

1.55 

2.63 

3.02 

1.33 

1.56 

1.85 

2.28 

|gg| 

Total  Number  of  Entries 

280.17 

55.79 

314.20 

90.00 

295.64 

71.50 

44.67 

26.45 

75.20 

52.57 

58.54 

41.34 

162.42  129.83 

194.70  143.86 

177.09  134.05 

HV 

Flight  Plan  Readouts 

9.67 

7.92 

3.40 

2.07 

6.82 

6.69 

0.00 

0.00 

0.00 

m 

0.00 

0.00 

4.83 

7.35 

1.70 

2.50 

3.41 

5.79 

LJ.V 

J-rings 

3.00 

4.38 

3.60 

4.39 

3.27 

4.17 

0.00 

0.00 

1.00 

1.73 

0.45 

1.21 

1.50 

3.34 

2.30 

3.43 

5.79 

1.86 

: 

Route  Displays 

1.67 

2.66 

2.00 

2.92 

1.82 

2.64 

0.17 

0.41 

1.34 

0.36 

0.92 

0.92 

1.98 

1.30 

2.26 

1.09 

2.07 

Total  Number  of  Entries 

274.38 

41.57 

330.25 

67.94 

302.31 

62.25 

60.63 

51.85 

97.19 

51.05 

78.91 

53.92 

167.50  118.02 

213.72  132.33 

190.61 

126.54 

b 

Flight  Plan  Readouts 

8.00 

6.35 

5.56 

5.33 

6.78 

5.90 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

4.00 

6.00 

2.78 

4.66 

3.39 

5.37 

pC- 

J-rings 

2.56 

3.31 

2.88 

2.92 

2.72 

3.07 

0.31 

0.79 

0.69 

1.25 

0.50 

1.05 

1.44 

2.63 

1.78 

2.47 

1.61 

2.54 

Route  Displays 

2.50 

2.78 

1.38 

1.93 

1.94 

2.42 

0.81 

1.56 

0.75 

1.18 

0.78 

1.36 

1.66 

2.38 

1.06 

1.61 

1.36 

2.03 

Table  BB-2.  ATC  Control  Actions:  Means  and  Standard  Deviations  (number). 


(Hard  Altitude 
I  Interim  Altitude 
(Removal  of  Interim  Altitude] 
I  Route  Change 


Hard  Altitude 
Interim  Altitude 
Removal  of  Interim  Altitude] 
Route  Change _ 


1.25 
14.75 

9.75 

5.25 


2.50| 

10.81 

5.68 

3.86! 


1.50 

28.33 

4.33 

3.00 


1.52 
7.31 
1.97 

3.52 


1.40 

22.90 

6.50 

3.90 


1.84 

10.86 

4.55 

3.63 


0.25 

1.25 

0.00 

1.75 


0.50 

1.89 

0.00 

2.87 


2.17 

2.17 

0.83 

2.50 


2.79 

3.37 

0.98 

3.89 


1.40 

1.80 

0.50 

2.20 


2.32| 

2.78 

0.85 

3.361 


0.75 

8.00 

4.88 

3.50 


1.75 

10.18 

6.40 

3.66 


1.83 

15.25 

2.58 

2.75 


2.17 

14.70| 

2.35 

3.55 


1.40 

12.35 

3.50 

3.05 


2.04 

13.29| 

4.43 

3.521 


|  Hard  Altitude 

vl 

I  Interim  Altitude 
|  Removal  of  Interim  Altitude] 
|  Route  Change 


2.33 

18.33 

3.00 

3.83 


1.86 

6.74 

1.67 

3.97 


3.80 

28.20 

5.20 

1.00 


4.66 

12.51 

2.68 

1.41 


3.00 

22.82 

4.00 

2.55 


3.32 

10.58 

2.37 

3.30 


0.50 

1.83 

0.67 

2.67 


0.84 

1.60] 

1.21 

3.081 


2.00 

6.20 

0.60 

0.60 


1.00 

3.90 

0.89 

1.34 


1.18 

3.82 

0.64 

1.73 


1.17 

3.54| 

1.03 

2.57| 


1.42 

10.08 

1.83 

3.25 


1.68 

9.80 

1.85 

3.44 


2.90 

17.20 

2.90 

0.80 


3.31 

14.52| 

3.07 

1.321 


2.09 

13.32 

2.32 

2.14 


2.60 

12.40 

2.48 

2.92 


j  Hard  Altitude 
|  Interim  Altitude 
I  Removal  of  Interim  Altitude] 
1  Route  Change _ 


1.44 

17.38 

4.50 

3.69 


1.86 

7.76 

4.24 

3.30 


3.44 

27.69 

4.69 

2.13 


5.27 

8.85 

2.36| 

2.73 


2.44 

22.53 

4.59 

2.91 


4.02 

9.72 

3.38 

3.08 


0.69 

2.56 

0.63 

2.63 


1.01 

3.03 

1.20 

2.83 


2.13 

5.19 

0.69 

1.81 


1.89] 

6.05 

0.79| 

2.93 


1.41 

3.88 

0.66 

2.22 


1.66 

4.89 

1.00 

2.86 


1.06 

9.97 

2.56 

3.16 


1.52 

9.50 

3.65 

3.07 


2.78 

16.44 

2.69 

1.97 


3.95 

13.65 

2.67 

2.79 


1.92 

13.20 

2.63 

2.56 


3.09| 

12.11 

3.171 

2.97 


Table  BB-3.  Station  Keeping  Activities:  Means  and  Standard  Deviations  (number). 


House  Keeping 
Activities 


IHandoffs  Initiated 
Handoffs  Accepted 
Datablock  Movements 
Handoffs  Initiated 
Handoffs  Accepted 
Datablock  Movements 
Handoffs  Initiated 
Handoffs  Accepted 
Datablock  Movements 
Handoffs  Initiated 
Handoffs  Accepted 
Datablock  Movements 


aamasag 

1  High  Load  1 

17.80 

19.80 

29.20 

12.17 

63.20 

54.50 

24.00 

18.75 

26.83 

15.38 

31.17 

15.05 

12.80 

13.61 

19.40 

16.52 

29.40 

24.40 

18.56 

17.15 

25.25 

14.45 

40.63 

35.69 

Load/Pos. 

Collapsed 


Table  BB-4.  Total  Number  of  Entries:  ANOVA  Results. 


Position 


MS  Effect  MS  Error  I  F 
793291.563  2862.092  277.172" 


1  df  1  df2  /7-level 


SB 

ad 

35588.352  2862. 

[HKI 

43 

KB 

52 

ABES 

EE 

S3 

tomation 
sition  X  Load 

4311.262  2862. 
722.574  2862. 

^■ETl 

50 

25 

KB 

HI 

£ 

m 

Mgr 

HE 

Load  X  Automation 


6471.628 

1428.460 


1|  1 1»  li|  HMilM  I II 


Position  X  Load  X  Automation  826.108  2862.092  0.289  2  52  .750 

Table  BB-5.  Information  Pickup:  MANOVA  Results. 


Ro/s  |  F  df  1  df  2  /-level 


0.515  1.062  1.062  17.705  • 


Wilks  A 

Position 

.485 

_ 

.891 

0.111  0.121  0.109  0.993 

position  X  Load  X  Automation  ,943|  0.057 

Table  BB-6.  Flight  Plan  Readout:  ANOVA  Results. 


Load 


Alternation 
Position  X  Load 
tesition  X  Automation 
Load  X  Automation 
Position  X  Load  X  Automation 


1  52 

21.815  18.410  1.185 

1  52 

18.410 


0.105  18.4101  0.006 


7.623  18.410  0.957 
7.623  18.410  0.957 


Table  BB-7.  J-Rings:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

dfl 

df  2 

/?-level 

Position 

WMMm 

■MEEK 

1 

52 

.001 

Load 

m 

52 

.523 

Automation 

1 

m 

52 

.754 

Position  X  Load 

.979 

Position  X  Automation 

|  ^ 

Ml 

52 

.771 

Load  X  Automation 

■■H 

m 

52 

.742 

Position  X  Load  X  Automation 

0.259 

6.047 

0.043 

2 

52 

.958 

Table  BB-8.  Route  Display:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

ira i 

18(4^41 

Position 

23.488 

3.943 

5.957 

i 

Load 

8.102 

3.943 

2.055 

m 

K 

.158 

Automation 

^■ESZsu 

3.943 

Position  X  Load 

3.943 

EPdcE 

i 

Position  X  Automation 

■HaEfi 

.  2 

52 

.743 

Load  X  Automation 

4.866 

i 

K 

B 

Table  BB-9.  ATC  Control  Actions:  MANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

F 

dfl 

df  2 

p-  level 

Position 

.230 

0.770 

3.351 

3.351 

4 

■S 

.000 

Load 

■EiS 

0.676 

mm 

.000 

Automation 

mmm 

■K1EEB 

HKB 

mm 

BEH 

Kjjjj 

.127 

Position  X  Load 

gjjjgg 

K 

■EE 

■Via 

Position  X  Automation 

HK 

■wag 

8 

98 

Load  X  Automation 

■till 

8 

98 

Position  X  Load  X  Automation 

.756 

0.249 

0.317 

0.296 

1.839 

8 

B 

.079 

Table  BB-10.  Hard  Altitudes:  ANOVA  Results. 


Qg| 

fraa 

/7-level 

Position 

m 

52 

.155 

Load 

51.657 

9.301 

5.554 

1 

52 

.022 

Automation 

BBMEE 

9.301 

0.693 

2 

52 

.505 

Position  X  Load 

9.301 

0.187 

1 

52 

.667 

Position  X  Automaton 

3.642 

9.301 

0.392 

2 

52 

.678 

Load  X  Automation 

■H 

9.301 

0.504 

2 

52 

.607 

Position  X  Load  X  Automation 

9.301 

1.060 

2 

52 

.354 

Table  BB-1 1 .  Route  Changes:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

dfl 

df  2 

/7-level 

Position 

10.439 

1 

52 

.292 

Load 

mmm 

1 

52 

.102 

Automation 

WlfPw 

2 

52 

.515 

Position  X  Load 

1 

52 

.503 

Position  X  Automation 

.475 

Load  X  Automation 

5.545 

9.198 

0.603 

.551 

Position  X  Load  X  Automation 

9.198 

0.470 

2 

52 

.627 

BB-3 


Table  BB-12.  Interim  Altitudes:  ANOVA  Results. 


Position 

Load 

Automation 
Position  X  Load 

Position  X  Automation _ 

Load  X  Automation 
Position  X  Load  X  Automation 


MS  Effect 


MS  Error 


5334.458 

716.678 

35.445 

225.357 

18.928 

2.786 

24.448 


49.944 

49.944 

49.944 

49.944 

49.944 

49.944 

49.944 


F 

106.808' 

14.350' 

0.710' 

4.512' 

0.379' 

0.056' 

0.490' 


df  1 
1 

1 

2 

1 

2 

2 

2 


df  2 
52 
52 
52 
52 
52 
52 
52 


p-level 

^00 

.000 

.497 

.038 

.686 

.946 

.616 


Table  BB-13.  Removal  of  Interim  Altitudes:  ANOVA  Results. 


F 

Bn 

on 

Position 

286.797 

■i 

B1 

Load 

0.245 

4.464 

0.055 

1 

52 

.816 

Automation 

2 

m 

MBfsgE 

Position  X  Load 

BOSS! 

KQE! 

1 

m 

■ES 

Position  X  Automation 

21.312 

4.464 

mm 

m 

Position  X  Load  X  Automation 

31.075 

m 

■ 

Table  BB-14.  Station  Keeping:  MANOVA  Results. 


Position 

Load 

Automation 
Position  X  Load 

Position  X  Automation _ 

Load  X  Automation 
Position  X  Load  X  Automation 


Wilks  A 
.190 
.773 
.910 
.888 
.887 
.963 
.904 


Pillai’s 

0.810 

0.227 

0.090 

0.112 

0.113 

0.037 

0.097 


Hotelling 


Roy’s 


4.257^  4.257 


0.293 


0.293 


0.099 


0.098 


0.127 


0.127 


0.127 

0.125 


0.038 


0.028 


0.10$ 


0.101 


F 

70.951 

4.887 

0.808 

2.110 

1.032 

0.319' 

0.864' 


df  1 


df  2 


59 


50 


a  109 


50 


100 


100 


109 


/7-level 

XXK) 

.005 

.566 

.111 

.409 

.926 

.524 


TableBB-15.  Handoffs  Initiated:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

BP 

Position 

27021.953 

350.172 

77.168 

m 

m 

■ 

Load 

1 

m 

.243 

Automation 

14.636 

350.172 

WMm, 

wm 

Position  X  Load 

1.257 

350.172 

g>M8Ei 

m 

Position  X  Automation 

267.619 

350.172 

0.764 

m 

Load  X  Automation 

36.886 

350.172 

0.105 

2 

MS 

Position  X  Load  X  Automation 

664.876 

350.172 

1.899 

2 

52 

.160 

Table  BB-16.  Handoffs  Accepted:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

df  1 

df2 

/Mevel 

Position 

60480.059 

329.186 

183.726 

1 

52 

.000 

Load 

4823.961 

329.186 

14.654 

1 

52 

.000 

Automation 

aai 

52 

.300 

Position  X  Load 

1 

52 

.068 

Position  X  Automation 

329.186 

0.131 

2 

52 

.877 

Load  X  Automation 

329.186 

0.354 

2 

52 

.703 

329.186 

52 

.531 

Table  BB-17.  Datablock  Movements:  ANOVA  Results. 


cm 

E2 

Position 

53321.758 

893.799 

1 

.000 

Load 

2798.864 

893.799 

iff] 

mi 

.083 

Automation 

1666.376 

893.799 

m 

52 

.165 

Position  X  Load 

925.341 

893.799 

1 

52 

.314 

Position  X  Automation 

2327.010 

2 

52 

.084 

Load  X  Automation 

630.010 

893.799 

0.705 

2 

52 

.499 

Position  X  Load  X  Automation 

318.709 

893.799 

0.357 

2 

52 

.702 

BB-5 


APPENDIX  CC 


Entry  Questionnaire  Results 


Table  CC  -1.  General  Background  Questions. 


Means 

SDs 

Use  of  vectoring  for  separation. 

6.25 

1.44 

Use  of  vertical  separation. 

WmM 

1.61 

Use  of  speed  control  for  separation. 

5.06 

1.95 

Experience  with  video  games. 

5.13 

3.01 

Table  CC-2.  Importance  of  Aircraft  Information. 


Variable  Label 

Means 

SDs 

Current  Aircraft  Location 

1.71 

Current  Altitude 

1.57 

Arrival  Airport  (within  sector) 

8.13 

1.71 

Most  Recently  Assigned  Altitude 

7.81 

2.23 

Current  Heading 

7.67 

1.50 

Density  of  Aircraft  on  Radar  Display 

7.63 

1.45 

Most  Recently  Assigned  Heading 

7.47 

2.00 

Aircraft  Type 

7.44 

2.19 

Entry  Airspeed 

7.31 

2.69 

Aircraft  CallSign 

7.19 

3.45 

Current  Airspeed 

7.19 

2.04 

Aircraft  Waiting  for  Hand-off/Release 

7.13 

1.15 

Most  Recently  Assigned  Airspeed 

6.75 

2.08 

Aircraft  Holding/Spinning 

6.75 

2.38 

Aircraft  Near  Exit  Fix/ Arrival  Airport 

6.69 

2.15 

Controller  Ownership 

6.50 

1.86 

Entry  Altitude 

6.50 

2.55 

Departure  Airport  (within  sector) 

5.69 

2.65 

Entry  Fix 

5.06 

2.77 

Exit  Altitude 

4.94 

2.59 

Exit  Fix 

4.56 

2.90 

Exit  Airspeed 

4.19 

2.40 

Aircraft  Beacon  Code 

3.13 

1.82 

CC-1 


Table  CC-3.  Importance  of  Radarscope  Information. 


Variable  Label 

Means 

SDs 

Sector  Boundaries 

8.25 

1.29 

Restricted  Area  Boundaries 

7.88 

1.26 

Filter  Settings 

Collision  Alert 

6.56 

KElifl3l 

ILS  Approaches 

5.19 

Obstructions 

■aw« 

Airports 

mmm 

VORs 

1.92 

Fixes 

mm 

Holding  Patterns 

warn 

System  Clock 

■ 

Future  Aircraft  List 

■&a 

CC-2 


APPENDIX  DD 


Post-Scenario  Questionnaire  Results 


Table  DD-1.  Post-Scenario  Questionnaire:  Means  and  Standard  Deviations. 


General  Questions 

■  m 

sms 

BBSS! 

in 

Ilf? 

SiM 

KM 

1 11 

Load/Pos. 

Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

REM 

REM 

IBTiffHKEM 

Ba»M 

REM 

REM 

mm 

Simulation  Realism 

6.88 

1.63 

5.75 

2.11 

6.31 

1.94 

6.69 

1.96 

5.31 

1.92 

2.03 

6.78 

1.77 

5.53 

2.00 

6.16 

1.98 

-A-'-'v"  ill 

Representation  of  Typical  Workday 

6.76 

1.51 

5.26 

2.32 

6.01 

2.07 

6.44 

1.95 

4.76 

1.98 

5.60 

2.12 

1.73 

5.01 

2.13 

5.80 

2.09 

sii 

How  Well  You  Controlled  Traffic 

8.00 

1.27 

6.62 

1.78 

7.31 

1.67 

7.19 

2.37 

6.12 

1.75 

6.66 

2.12 

7.59 

1.92 

6.37 

1.75 

6.98 

1.92 

mmM 

Scenario  Difficulty 

6.42 

2.83 

5.35 

2.41 

5.88 

2.64 

6.92 

2.73 

5.86 

2.09 

6.39 

2.45 

6.67 

2.75 

5.60 

2.24 

6.13 

2.54 

ATWIT  Device  Interference 

2.12 

1.30 

2.22 

1.26 

2.17 

1.26 

2.59 

1.39 

2.68 

1.60 

2.64 

1.48 

2.36 

1.35 

2.45 

1.44 

1.38 

'  j 

Oculometer  Interference 

2.17 

0.94 

2.41 

0.84 

2.29 

0.88 

2.22 

3.29 

2.12 

3.29 

2.14 

2.73 

1.77 

2.85 

1.65 

2.79 

1.70 

Simulation-Pilot  Respondence 

7.45 

2.09 

6.64 

2.28 

7.05 

2.19 

2.17 

6.58 

1.90 

6.89 

2.03 

7.33 

2.10 

6.61 

2.07 

6.97 

vm 

DST  Effectiveness  Using  TFPs 

XXX  XXX 

XXX  XXX 

XXX  XXX 

XXX  XXX 

Competence  Using  DST 

XXX  XXX 

XXX  XXX 

XXX  XXX 

XXX  XXX 

Klii® 

Simulation  Realism 

6.19 

1.76 

5.82 

1.72 

1.72 

6.50 

1.71 

6.32 

1.62 

6.41 

1.64 

6.35 

1.71 

6.07 

1.66 

6.21 

1.68 

Representation  of  Typical  Workday 

5.85 

1.88 

5.20 

1.82 

5.52 

1.85 

5.98 

1.82 

5.57 

1.69 

5.77 

1.74 

5.91 

1.82 

5.39 

1.74 

5.65 

1.79 

n 

How  Well  You  Controlled  Traffic 

8.19 

1.52 

7.19 

1.22 

7.69 

1.45 

7.19 

2.37 

6.94 

1.61 

7.06 

2.00 

7.69 

2.02 

7.06 

1.41 

7.37 

1.76 

item 

Scenario  Difficulty 

6.60 

3.17 

2.18 

2.74 

7.16 

2.42 

5.52 

1.85 

6.34 

2.28 

6.88 

2.79 

5.46 

1.99 

6.17 

2.51 

ATWIT  Device  Interference 

1.78 

1.00 

2.84 

1.48 

2.31 

1.35 

2.09 

1.33 

2.37 

1.23 

2.23 

1.27 

1.94 

1.17 

2.61 

1.36 

2.27 

1.30 

Ifsll 

Oculometer  Interference 

2.29 

0.90 

2.40 

0.84 

2.35 

3.79 

2.77 

3.58 

2.43 

3.69 

2.57 

3.04 

2.16 

2.99 

1.89 

3.02 

2.01 

Simulation-Pilot  Respondence 

7.84 

1.59 

6.78 

1.78 

7.31 

1.74 

7.84 

1.50 

7.29 

1.76 

7.57 

1.64 

7.84 

1.52 

7.04 

1.76 

7.44 

1.68 

DST  Effectiveness  Using  TFPs 

4.79 

1.30 

4.81 

1.31 

mSM 

1.28 

4.52 

2,21 

4.62 

1.99 

4.57 

2.07 

4.66 

1.79 

4.71 

1.66 

4.68 

1.71 

Bill 

Competence  Using  DST 

5.44 

1.66 

5.35 

5.40 

1.84 

5.96 

2.61 

5.39 

2.35 

5.68 

2.46 

5.70 

2.16 

5.37 

2.17 

5.54 

2.16 

Simulation  Realism 

5.63 

2.00 

5.00 

2.22 

5.31 

6.25 

2.14 

5.63 

2.00 

5.94 

2.06 

5.94 

2.06 

5.31 

5.63 

Baal 

: 

Representation  of  Typical  Workday 

5.16 

2.15 

4.63 

2.15 

4.90 

2.13 

6.01 

2.36 

4.76 

2.01 

5.38 

2.25 

5.58 

2.26 

4.69 

5.14 

2.19 

n  ■ 

How  Well  You  Controlled  Traffic 

7.94 

1.06 

5.94 

6.94 

7.37 

1.71 

5.69 

1.96 

6.53 

2.00 

7.66 

1.43 

5.81 

1.97 

6.73 

1.95 

|  , 

Scenario  Difficulty 

5.97 

2.90 

4.42 

2.74 

5.19 

2.89 

5.92 

3.02 

4.61 

2.69 

5.26 

2.89 

5.94 

2.91 

4.51 

2.68 

5.23 

2.87 

1 

ATWIT  Device  Interference 

1.87 

0.76 

2.99 

2.97 

2.42 

2.06 

1.33 

3.19 

2.03 

2.62 

1.78 

1.97 

1.07 

3.62 

2.56 

2.80 

2.12 

Oculometer  Interference 

2.47 

0.76 

2.29 

2.38 

3.41 

2.17 

3.61 

2.73 

3.51 

2.43 

2.94 

1.67 

2.95 

2.11 

2.94 

1.89 

j  Sr* 

Simulation-Pilot  Respondence 

7.59 

1.54 

6.33 

2.19 

6.96 

1.97 

7.59 

1.49 

6.45 

2.10 

7.02 

1.88 

7.59 

1.49 

6.39 

2.11 

6.99 

1.91 

DST  Effectiveness  Using  TFPs 

5.10 

1.44 

5.14 

¥W\ 

5.12 

1.73 

7.06 

2.23 

6.12 

1.49 

6.59 

1.93 

6.08 

2.10 

5.63 

1.82 

5.85 

1.96 

f: 

Competence  Using  DST 

5.29 

1.84 

4.73 

1.87 

1.85 

6.40 

2.40 

5.24 

2.14 

5.82 

2.31 

5.84 

2.18 

4.99 

1.99 

5.42 

2.12 

-%*&.•  i 

Simulation  Realism 

6.23 

1.84 

5.52 

2.02 

5.88 

1.95 

6.48 

1.91 

5.75 

1.86 

6.12 

1.91 

6.36 

1.87 

5.64 

1.94 

mi 

1.93 

i 

Representation  of  Typical  Workday 

5.92 

1.94 

2.08 

5.48 

6.14 

1.90 

5.58 

1.98 

5.03 

1.98 

5.53 

•  5*  ; 

How  Well  You  Controlled  Traffic 

8.04 

1.27 

6.58 

1.76 

7.31 

1:69 

7.25 

2.13 

6.25 

1.82 

6.75 

7.64 

1.79 

6.42 

1.79 

1.89 

igB| 

Scenario  Difficulty 

6.33 

2.92 

2.45 

5.69 

2.75 

6.66 

2.73 

5.33 

2.26 

2.58 

6.50 

2.81 

5.19 

2.35 

5.84 

2.67 

ATWIT  Device  Interference 

1.92 

1.03 

2.16 

2.48 

1.77 

2.25 

1.34 

2.75 

1.65 

1.52 

2.09 

Ugy 

2.89 

1.92 

2.49 

1.65 

j  JEft 

Oculometer  Interference 

2.31 

0.86 

2.37 

0.84 

2.34 

2.36 

3.49 

2.39 

2.36 

1.86 

2.93 

1.87 

2.92 

1.86 

Simulation-Pilot  Respondence 

7.63 

1.73 

6.59 

2.06 

7.11 

1.96 

7.54 

1.73 

6.78 

1.92 

7.16 

1.86 

7.59 

1.72 

6.68 

1.98 

7.13 

1.91 

DST  Effectiveness  Using  TFPs 

5.06 

1.11 

5.08 

1.38 

1.25 

5.63 

m 

5.35 

1.54 

5.49 

1.82 

5.34 

1.68 

5.21 

1.46 

5.28 

1.57 

Competence  Using  DST 

5.40 

1.40 

5.18 

1.60 

5.29 

1.50 

5.94 

ns 

5.37 

1.79 

5.65 

1.93 

5.67 

1.76 

5.27 

1.69 

5.47 

1.74 
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Table  DD-2.  Realism:  MANOV A  Results. 


Table  DD-3.  Realism:  ANOVA  Results. 


Table  DD-4.  Representativeness:  ANOVA  Results. 


MS  Effect 

MS  Error 

F 

BUI 

p-level 

Position 

0.570 

_ 

mm2 

Load 

48.459 

1 

me 

BESS 

Automation 

7.728 

1.514 

.012 

Position  X  Load 

km 

mostE 

mm 

S 

ME 

KE 

Position  X  Automation 

— hrse 

2 

■tfii 

mmm 

Load  X  Automation 

wmbm 

2.156 

m 

18! 

n 

Position  X  Load  X  Automation 

0.936 

2.072 

0.452 

2 

30 

.641 

Table  DD-5.  Participant  Performance:  ANOVA  Results. 


MS  Effect 

MS  Error 

P 

EBI 

BTH 

E3EBI 

Position 

15.188 

3.008 

mm 

15 

Load 

72.535 

2.010 

36.080 

1 

15 

BBSS 

■bh 

euae 

2 

18! 

Position  X  Load 

2.518 

2.120 

m 

ME 

.293 

I H2EE 

0.159 

18! 

.854 

Load  X  Automation 

5.939 

1.983 

2.995 

2 

30 

.065 

Position  X  Load  X  Automation 

0.256 

3.173 

0.081 

2 

30 

.923 
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Table  DD-6.  Difficulty:  ANOVA  Results. 


Position _ 

Load _ 

Automation _ 

Position  X  Load _ 

Position  X  Automation _ 

Load  X  Automation _ 

Position  X  Load  X  Automation 


MS  Effect 
4.477' 
81.795' 
18.212' 
0.047' 
0.783' 
0.680' 
0.475' 


MS  Error 


df  1 


df  2 


2.972 


1.506 


15 


33.430  2.447 


15 


4.510  4.038 


2  30 


4.253 


0.011 


1 


15 


3.245 


0.241 


30 


4.804  0.142 


30 


6.114 


0.078 


30 


jo-level 

239 

.139 

.028 

.918 

.787 

.869 

.925 


Table  DD-7.  Difficulty:  ANOVA  Results  (N=6). 


Position _ 

Load _ 

Automation _ 

Position  X  Load _ 

Position  X  Automation _ 

Load  X  Automation _ 

Position  X  Load  X  Automation 


MS  Effect 
4.014 
21.125 
7.681 
6.125 
1.347 
1.042 
1.542 


MS  Error 
4.847 
12.758 
3.747 
7.492 
7.381 
7.475 
4.408 


F 

0.828' 

16.470' 

2.050' 

0.818' 

0.183' 

0.139' 

0.350' 


df  1 


df  2 


10 


10 


10 


10 


/7-level 

405 

.010 

.179 

.407 

.836 

.872 

.713 


Table  DD-8.  AT  WIT  Interference:  ANOVA  Results. 


Position 


Load 


Automation 


(Position  X  Load 


[Position  X  Automation 


Load  X  Automation 


[Position  X  Load  X  Automation 


MS  Effect 


0.012 


31.254 


4.753 


4.536 


2.736 


1.006 


1.214 


MS  Error 


2.163 


2.299 


1.505 


2.258 


1.423 


2.027 


1.629 


0.006  1 


13.594  1 


3.158 


2.009  1 


1.923 


4.936 


0.746 


df  1 


df  2 


15 


15 


30 


15 


30 


30 


/>-level 


.942 


.002 


.057 


.177 


.164| 


30  .014 


.483 


Table  DD-9.  AT  WIT  Interference:  ANOVA  Results  (N~6). 


Position 


Load 


Low  Load 


High  Load 


D  Side 


RSide 


No  Automation 


Limited  Automation 


Full  Automation 


Automation 


Low  Load 


High  Load 


[Position  x  Load 


Position  x  Automation 


Load  x  Automation 


D  side 


R  side 


Position  x  Load  x  Automation 


MS  Effect 


6.413 


0.593 


8.051 


12.077 


0.148 


1.384 


0.131 


0.253 


16.936 


2.936 


0.698 


6.411 


19.519 


6.505 


11.282 


0.694 


15.475 


4.887 


MS  Error 


0.437 


0.215 


0.359 


0.371 


0.148 


0.404 


0.056 


0.329 


0.595 


1.827 


0.681 


2.683 


0.494 


1.1731 


0.873 


14.668 


2.759 


22.426 


32.545 


1.000 


25.725 


2.353 


0.770 


28.465 


1.607 


0.630  1.109 


9.414 


1.284  15.198 


2.425 


0.794  14.203 


1.404 


13.197 


5.600 


df  1 


df  2 


10 


10 


10 


10 


10 


2  10 


10 


10 


^-level 


.012 


.158 


.005 


.002 


.363 


.004 


.186 


.420 


.003 


.248 


.367 


.005 


.011 


.139 


.001 


.290 


.002 


.023 


DD-3 


Table  DD-10.  Oculometer  Interference:  ANOV A  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

0.001 

1.263 

0.001 

1 

15 

.982 

Automation 

1.248 

1.459 

0.856 

2 

30 

.435 

Load  X  Automation 

0.319 

3.608 

0.088 

2 

30 

.916 

Table  DD-11.  Responsiveness  of  Simulation  Pilots:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df2 

p-level 

Position 

0.137 

1.385 

0.099 

1 

15 

.757 

Load 

39.298 

2.732 

14.383 

1 

15 

.002 

Automation 

4.485 

2.117 

2.119 

2 

30 

.138 

Position  X  Load 

0.899 

1.395 

0.644 

1 

15 

.435 

Position  X  Automation 

0.675 

1.580 

0.427 

2 

30 

.656 

Load  X  Automation 

1.044 

2.530 

0.413 

2 

30 

.666 

Position  X  Load  X  Automation 

0.169 

2.177 

0.078 

2 

30 

.925 

Table  DD-12.  Effectiveness  of  the  DST  in  Resolving  Conflicts  When  Using  Trial  Flight 

Plans:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df  2 

p-level 

Position 

34.406 

2.102 

16.368 

1 

15 

.001 

Load 

3.194 

1,625 

1.966 

1 

15 

.181 

Position  X  Load 

3.852 

3.261 

1.181 

1 

15 

.294 

Table  DD-13.  Competence  Felt  Using  the  DST:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

dfl 

df  2 

p-level 

Position 

9.595 

8.146 

1.178 

1 

15 

.295 

Load 

11.234 

1.207 

9.307 

1 

15 

.008 

Automation 

0.470 

1.330 

0.354 

1 

15 

.561 

Position  X  Load 

2.301 

1.821 

1.263 

1 

15 

.279 

Position  X  Automation 

2.258 

1.819 

1.241 

1 

15 

.283 

Load  X  Automation 

2.215 

1.992 

1.112 

1 

15 

.308 

Position  X  Load  X  Automation 

0.029 

2.016 

0.014 

1 

15 

.906 
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APPENDIX  EE 


Subject  Matter  Expert  Rating  Forms/OTS  Results 


Table  EE-1.  Maintaining  Safe  and  Efficient  Traffic  Flow:  Means  and  Standard 

Deviations. 


Safety  anc 
Efficiency 

Question 

-  -  \*S3K§ 

C’ontri 

HQ 

ty>inou 

f&silfplf 

cm 

m 

c#.#aN 

Load/Pos. 

Collapsed 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Means)  SDs 

Means) 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means  | 

SDs 

Means) 

SDs 

Means)  SDs 

Means) 

SDs 

1)  Separation  and  Resolving  Potential  Conflicts 

5.92 

1.48 

5.37 

1.70 

5.64 

1.60 

5.92 

1.48 

5.37 

1.70 

5.64 

1.60 

5.92 

1.46 

5.37 

1.68 

5.64 

1.58 

2)  Sequencing  Arrival/Departure  Aircraft 

6.41 

1.30 

5.79 

1.66 

6.10 

1.50 

6.41 

1.30 

5.79 

1.66 

6.10 

1.50 

6.41 

1.28 

5.79 

1.64 

6.10 

1.49 

3)  Using  Control  Instructions 

6.47 

0.88 

5.72 

1.36 

6,09 

1.19 

6.47 

0.88 

5.72 

1.36 

6.09 

1.19 

6.47 

0.87 

5.72 

1.34 

6.09 

1.18 

1 )  Separation  and  Resolving  Potential  Conflicts 

6.24 

0.46 

5.92 

1.29 

6.08 

0.96 

6.24 

0.46 

5.92 

1.29 

6.08 

0.96 

6.24 

0.45 

5.92 

1.27 

0.96 

IKirt^p 

2)  Sequencing  Arrival/Departure  Aircraft 

6.37 

0.58 

6.45 

0.69 

6.41 

0.63 

6.37 

0.58 

6.45 

0.69 

6.41 

0.63 

6.37 

0.57 

6.45 

0.68 

6.41 

0.62 

g&  ■?»-*&  tv  ■- 

3)  Using  Control  Instructions 

6.30 

0.67 

6.25 

0.97 

6.27 

0.82 

6.30 

0.67 

6.25 

0.97 

6.27 

0.82 

6.30 

0.66 

6.25 

0.96 

6.27 

0.82 

1 )  Separation  and  Resolving  Potential  Conflicts 

6.36 

0.89 

5.74 

1.34 

6.05 

1.16 

6.36 

0.89 

5.74 

1.34 

6.05 

1.16 

6.36 

0.88 

5.74 

1.32 

6.05 

1.15 

2)  Sequencing  Arrival/Departure  Aircraft 

6.68 

0.57 

6.10 

1.44 

6.39 

1.12 

6.68 

0.57 

6.10 

1.44 

6.39 

1.12 

6.68 

0.56 

6.10 

1.41 

6.39 

1.11 

3)  Using  Control  Instructions 

6.48 

0.60 

6.28 

0.85 

6.38 

0.73 

6.48 

0.60 

6.28 

0.85 

6.38 

0.73 

6.48 

6.28 

0.84 

6.38 

0.73 

1)  Separation  and  Resolving  Potential  Conflicts 

6.17 

1.03 

5.67 

1.44 

5.92 

1.27 

6.17 

1.03 

5.67 

1.44 

5.92 

1.27 

6.17 

1.02 

5.67 

1.44 

5.92 

1.27 

2)  Sequencing  Arrival/Departure  Aircraft 

6.49 

0.88 

6.11 

1.33 

6.30 

1.14 

6.49 

0.88 

6.11 

1.33 

6.30 

1.14 

6.49 

0.87 

6.11 

1.32 

6.30 

1.13 

3)  Using  Control  Instructions 

6.42 

0.72 

6.08 

1.09 

6.25 

0.93 

6.42 

0.72 

6.08 

1.09 

6.25 

0.93 

6.42 

0.71 

6.08 

1.09 

6.25 

0.93 

Table  EE-2.  Prioritizing:  Means  and  Standard  Deviations. 


prioritizing 


Question 


8)  Taking  Actions  in  Appropriate  Order 

9)  Preplanning  Control  Actions 

10)  Handling  Control  Tasks  for  Several  Aircraft 

1 1 )  Marking  Flight  Strips  and  Performing  Tasks 


8)  Taking  Actions  in  Appropriate  Order 

9)  Preplanning  Control  Actions 

1 0)  Handling  Control  Tasks  for  Several  Aircraft 

11)  Marking  Flight  Strips  and  Performing  Tasks 


8)  Taking  Actions  in  Appropriate  Order 

9)  Preplanning  Control  Actions 

10)  Handling  Control  Tasks  for  Several  Aircraft 

11)  Marking  Flight  Strips  and  Performing  Tasks 


8)  Taking  Actions  in  Appropriate  Order 

9)  Preplanning  Control  Actions 

10)  Handling  Control  Tasks  for  Several  Aircraft 

1 1)  Marking  Flight  Strips  and  Performing  Tasks 


ism 

hmimi 1 

Load/Pos. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Collapsed 

Means)  SDs 

Means  |  SDs 

■am 

EHJI 

■area 

EE9 

1223331 

E3l 

1332311 

■am 

Means]  SDs 

6.07 

1.18 

5.45 

1.37 

5.76 

1.30 

6.07 

1.18 

5.45 

1.37 

5.76 

1.30 

6.07 

1.16 

5.45 

1.35 

5.76 

1.29 

4.91 

1.00 

4.10 

1.39 

4.50 

1.26 

4.91 

1.00 

4.10 

1.39 

4.50 

1.26 

4.91 

0.98 

4.10 

1.37 

4.50 

1.25 

6.30 

0.99 

5.66 

1.55 

5.98 

1.32 

6.30 

0.99 

5.66 

1.55 

5.98 

1.32 

6.30 

0.98 

5.66 

1.53 

5.98 

1.31 

3.62 

2.62 

3.12 

2.21 

3.37 

2.40 

3.62 

2.62 

3.12 

2.21 

3.37 

2.40 

3.62 

2.58 

3.12 

2.17 

3.37 

2.38 

6.26 

0.57 

6.21 

gj 

6.24 

0.55 

6.26 

0.57 

6.21 

0.54 

6.24 

0.55 

6.26 

0.56 

6.21 

0.53 

6.24 

0.54 

5.14 

1.06 

4.88 

5.01 

0.92 

5.14 

1.06 

4.88 

0.77 

5.01 

0.92 

5.14 

1.04 

4.88 

0.76 

5.01 

0.92 

6.30 

0.67 

6.12 

0.58 

6.21 

0.62 

6.30 

6.12 

0.58 

6.21 

0.62 

6.30 

0.66 

6.12 

0.57 

6.21 

0.62 

3.48 

0.00 

3.48 

itiBi 

3.48 

3.48 

3.48 

0.00 

3.48 

0.00 

3.48 

0.00 

3.48 

0.00 

3.48 

0.00 

6.51 

0.62 

5.95 

0.93 

6.23 

0.83 

6.51 

0.62 

5.95 

0.93 

6.23 

0.83 

6.51 

0.61 

5.95 

0.91 

6.23 

0.82 

5.21 

0.80 

4.35 

0.94 

4.78 

0.96 

5.21 

0.80 

4.35 

0.94 

4.78 

0.96 

5.21 

0.78 

4.35 

0.92 

4.78 

0.95 

6.73 

0.76 

6.34 

1.01 

6.54 

0.90 

6.73 

0.76 

6.34 

1.01 

6.54 

6.73 

0.74 

6.34 

0.99 

6.54 

0.89 

3.70 

0.88 

3.48 

0.00 

3.59 

0.62 

3.70 

0.88 

3.48 

0.00 

3.59 

0.62 

3.70 

0.87 

3.48 

0.00 

3.59 

0.62 

6.28 

0.84 

5.87 

1.03 

6.08 

0.96 

6.28 

0.84 

5.87 

1.03 

6.08 

0.96 

6.28 

0.84 

5.87 

1.03 

6.08 

5.09 

0.95 

4.44 

1.10 

4.76 

1.07 

5.09 

0.95 

4.44 

1.10 

4.76 

1.07 

5.09 

0.94 

4.44 

lira 

4.76 

1.07 

6.44 

0.83 

6.04 

1.13 

6.24 

1.01 

6.44 

0.83 

6.04 

1.13 

6.24 

1.01 

6.44 

0.82 

6.04 

1.13 

6.24 

tm 

3.60 

1.57 

3.36 

1.26 

3.48 

1.42 

3.60 

1.57 

3.36 

1.26 

3.48 

1.42 

3.60 

1.56 

3.36 

1.25 

3.48 

1.42 

Table  EE-3.  Providing  Control  Information:  Means  and  Standard  Deviations. 


12)  Providing  Essential  ATC  Information 

13)  Providing  Additional  ATC  Information 

14)  Providing  Coordination 

12)  Providing  Essential  ATC  Information 


Auto  2 


13)  Providing  Additional  ATC  Information 

14)  Providing  Coordination 


112)  Providing  Essential  ATC  Information 
13)  Providing  Additional  ATC  Information 
14)  Providing  Coordination 


Auto  j 
Collapsed! 


12)  Providing  Essential  ATC  Information 

13)  Providing  Additional  ATC  Information 

14)  Providing  Coordination 


5.34  1.07 
3.59  1.35 
5.92  0.84 


5.55  0.60 
4.57  1.29 
5.63  0.86 


5.73  0.55 
4,20  1.31 
6.07  1.27 


5.54  0.78 
4.12  1.35 
5.87  1.00 


5.19  0.62 
3.55  1.49 
5.43  1.06 


5.03 

3.53 

4.86 


1.32 

1.45| 

1.61 


4.96  1.11 
3.31  1.36 
4.86  1.56 


5.00  1.21 
3.21  1.25 
5,11  1.61 


5.37  0.63 
4.06  1.46 
5.53  0.95 


5.38  1.06 
3.86  1.40 
5.46  1.55 


5.25  1.00 
3.71  1.41 

5.37  1.40 


5.55  0.601 
4.57  1.29 
5.63  0.86 


5.73  0.55; 
4.20  1.31 
6.07  1.271 


5.54  0.78 
4.12  1.35 
5.87  1.001 


4.66  1.27| 
2.84  1.05 
4.30  1.80| 


5.19  0.62 
3.55  1.49 
5.43  1.06 


5.03  1.32 
3.53  1.45 
4.86  1.61 


4.96  1.11 
3.31  1.36 
4.86  1.56 


5.00  1.21 
3.21  1 .25| 
5.11  1.61 


5.37  0.63 
4.06  1.46| 
5.53  0.95 


5.38  1.06 
3.86  1.40 
5.46  1.55 


5.25  1 .00| 
3.71  1.41 

5.37  1.401 


5.34  1.05! 
3.59  1.33 
5.92  0.83 


5.55  0.59 
4.57  1.27 
5.63  0.84 


5.73  0.54 
4.20  1.29 
6.07  1.25 


5.54  0.77 
4.12  1.35 
5.87  1. 00 


4.66  1.25 
2.84  1.03 
4.30  1.77 


5.19  0.61 
3.55  1.46 
5.43  1.04 


5.03  1.30| 
3.53  1.43 
4.86  1.591 


4.96  1.11 
3.31  1.35 
4.86  1.55 


5.00  1 .20| 
3.21  1.24! 
5.11  1.601 


5.37  0.62 
4.06  1.45 
5.53  0.94 


5.38  1.05 
3.86  1.39 
5.46  1.54 


5.25  1.001 
3.71  1.40 
5.37  1.40 


Table  EE-4.  Technical  Knowledge:  Means  and  Standard  Deviations. 


Technical 

Question 

Mffl 

m 

S 

•p4& 

I 

Si 

Hil 

US 

HP 

Load/Pos. 

Knowledge 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

High  Load 

Load  Coll. 

Low  Load 

53 1 

j?  -f 

1 5)  Knowledge  of  LOAs  and  SOPs 

6.78 

0.65 

6.03 

1.32 

6.40 

1.09 

6.78 

0.65 

1.32 

itn 

EEE1 

mi 

-  AiltO-|'T| 

16)  Knowledge  of  Capabilities  and  Limitations 

6.72 

0.85 

6.35 

1.01 

6.53 

0.94 

6.72 

0.85 

6.35 

1.01 

6.53 

6.72 

0.84 

6.35 

6.53 

17)  Effective  Use  of  Equipment 

6.44 

0.73 

5.69 

1.40 

6.06 

1.16 

6.44 

0.73 

5.69 

1.40 

EH 

1.16 

6.44 

0.71 

5.69 

1  38 

6.06 

Hjffi 

TK 

1 5)  Knowledge  of  LOAs  and  SOPs 

6.88 

0.45 

6.93 

0.13 

6.90 

ESS 

6.88 

0.45 

6.93 

0.13 

6.90 

0.33 

6.88 

0.44 

6.93 

0.13 

6.90 

0.33 

Auto  2  . 

16)  Knowledge  of  Capabilities  and  Limitations 

6.71 

0.43 

7.00 

0.29 

6.85 

9 

6.71 

0.43 

7.00 

6.85 

0.39 

6.71 

0.42 

7.00 

0.28 

6.85 

0.39 

--JI 

17)  Effective  Use  of  Equipment 

6.00 

0.82 

5.69 

1.08 

5.85 

m 

0.82 

5.69 

1.08 

5.85 

0.95 

6.00 

0.80 

5.69 

1.06 

5.85 

0.95 

;  1  •  : 

1 5)  Knowledge  of  LOAs  and  SOPs 

6.82 

0.62 

6.78 

0.54 

6.80 

0.57 

6.82 

0.62 

6.78 

0.54 

6.80 

0.57 

6.82 

0.61 

6.78 

0.53 

6.80 

ISCT 

Auto  3 

1 6)  Knowledge  of  Capabilities  and  Limitations 

7.02 

0.28 

6.78 

0.65 

6.90 

0.51 

7.02 

0.28 

6.78 

0.65 

6.90 

0.51 

7.02 

0.27 

6.78 

0.64 

HsS 

v.~ 

1 7)  Effective  Use  of  Equipment 

6.50 

0.73 

5.75 

1.44 

6.13 

1.18 

6.50 

0.73 

5.75 

1.44 

6.13 

1.18 

6.50 

0.72 

5.75 

1.41 

6.13 

RHR 

AlltO 

1 5)  Knowledge  of  LOAs  and  SOPs 

6.83 

0.57 

6.58 

0.90 

0.76 

6.83 

0.57 

6.58 

0.90 

6.70 

0.76 

6.83 

0.57 

.  6.58 

0.90 

6.70 

0.76 

Collapsed 

16)  Knowledge  of  Capabilities  and  Limitations 

6.81 

0.58 

6.71 

0.75 

6.76 

0.67 

6.81 

0.58 

6.71 

0.75 

6  76 

0.67 

6.81 

0.58 

1  6.71 

0.75 

6.76 

0.67 

1 7)  Effective  Use  of  Equipment 

6.31 

0.78 

5.71 

1.29 

6.01 

1.10 

6.31 

0.78 

5.71 

1.29 

601 

1.10 

631 

0.77 

:  5.71 

1.28 

6.01 

1.10 

Table  EE-5.  Communicating:  Means  and  Standard  Deviations. 


Comm. 


Question 


1  8)  Using  Proper  Phraseology 

1 9)  Communicating  Clearly  and  Efficiently 

20)  Listening  for  Pilot  Readbacks  and  Requests 

1 8)  Using  Proper  Phraseology 

19)  Communicating  Clearly  and  Efficiently 

20)  Listening  for  Pilot  Readbacks  and  Requests 
1  8)  Using  Proper  Phraseology 

1 9)  Communicating  Clearly  and  Efficiently 

20)  Listening  for  Pilot  Readbacks  and  Requests 

1 8)  Using  Proper  Phraseology 

19)  Communicating  Clearly  and  Efficiently 

20)  Listening  for  Pilot  Readbacks  and  Requests 


6.77  0.65| 

6.78  0.65 
5.88  0.81 


6.63  0.58 
6.71  0.43 
6.19  0.54 


6.27  0.92 
6.22  1.22 
5.44  1.15 


6.84  0.45: 
6.99  0.47| 
6.19  0.75 


6.52  0.83 
6.50  1.00 
5.66  1.00 


6.73  0.52 
6.85  0.46 
6.19  0.64 


6.77  0.65 

6.78  0.65 
5.88  0.81 


6.63  0.58 
6.71  0.43 
6.19  0.54' 


6.27  0.92 
6.22  1.22 
5.44  1.15* 


6.84  0.45 
6.99  0.47 
6.19  0.75 


6.52  0.83 
6.50  1.00 
5.66  1.00 


6.73  0.52 
6.85  0.46 
6.19  0.64 


6.77  0.64 

6.78  0.64 
5.88  0.79; 


6.63  0.57 
6.71  0.42 
6.19  0.53 


6.27  0.91 
6.22  1.20 
5.44  1.13 


6.84  0.45 
6.99  0.46 
6.19  0.74 


6.88  0.45 
7.02  0.28 
6.25  0.58 


6.71  0.68 
6.91  0.45 
6.13  1.02 


6.79  0.57 
6.96  0.37 
6.19  0.82 


6.88  0.451 
7.02  0.28| 
6.25  0.58 


6.71  0.68 

6.91  0.45| 
6.13  1.02 


6.79  0.57 
6.96  0.37 
6.19  0.82 


6.88  0.44 
7.02  0.27 
6.25  0.57 


6.71  0.67 
6.91  0.44 
6.13  1.01 


6.76  0.57 
6.84  0.49 
6.11  0.66 


6.61  0.74 
6.71  0.85 
5.92  1.03 


6.68  0.66 
6.77  0.70 
6.01  0.86 


6.76  0.57 
6.84  0.49| 
6.11  0.66 


6.61  0.74 
6.71  0.85 
5.92  1.03 


6.68  0.66 
6.77  0.70 
6.01  0.86 


6.76  0.56] 
6.84  0.48 
6.11  0.66 


6.61  0.73 
6.71  0.85 
5.92  1.02 


Load/Pos. 
Collapsed 
Means  |  SDs 

6.52  0.82 
6.50  1.00 
5.66  1.00 
6.73  0.52 
6.85  0.46 
6.19  0.64 
6.79  0.57 
6.96  0.37 
6.19  0,31 
6.68  0.66 
6.77  0.69 
6.01  0.86 


EE-2 


Table  EE-6.  Maintaining  Safe  and  Efficient  Traffic  Flow:  MANOV A  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

F 

df  1 

df  2 

p-level 

Load 

■HUS 

warns, 

wm 

Automation 

immm 

mss 

MLLLU 

■ESE6H 

1.797 

6 

Load  X  Automation 

0.975 

38.269 

6 

2 

Table  EE  -7.  Maintaining  Separation  and  Resolving  Potential  Conflicts:  ANOVA  Results. 


mss 

[gOSEl 

F 

TTO 

p-level 

Load 

2.981 

WMOEM 

1 

m 

WKSm 

6.941 

2 

m 

Load  X  Automation 

0.102 

0.844 

0.121 

2 

14 

.887 

Table  EE  -8.  Sequencing  Arrival  and  Departure  Aircraft  Efficiently:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

1.688 

0.164 

1.309 

1 

7 

.015 

Automation 

0.481 

0.309 

1.558 

2 

14 

.245 

Load  X  Automation 

0.626 

0.605 

1.036 

2 

14 

.381 

Table  EE  -9.  Using  Control  Instructions  Effectively/Efficiently:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df2 

p-level 

Load 

1.333 

0.393 

3.394 

1 

7 

.108 

Automation 

0.341 

0.393 

0.867 

2 

14 

.442 

Load  X  Automation 

0.545 

0.517 

1.055 

2 

14 

.374 

Table  EE -10.  Prioritizing:  MANOV  A  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy's 

F 

df  1 

df  2 

p-level 

Load 

.173 

0.827 

Oj gg 

m 

5 

.024 

nj&OI 

m 

.107 

8.353 

8.353 

2.784 

6 

2 

.288 

Table  EE-1 1 .  Taking  Actions  in  an  Appropriate  Order  of  Importance:  ANOVA  Results. 


mom 

F 

Load 

2.083 

0.262 

7.955 

1 

7 

.026 

Automation 

RflMI 

2 

14 

.022 

Load  X  Automation 

2 

14 

.417 

Table  EE-12.  Preplanning  Control  Actions:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

1 

7 

Automation 

1.038 

0.386 

2.690 

m 

Load  X  Automation 

0.438 

0.640 

0.684 

2 

K 

Table  EE-13.  Handling  Control  Tasks  For  Several  Aircraft:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

so 

so 

Load 

■■H 

mss 

Wdril 

1 

7 

.028 

Automation 

OSS 

.117 

Load  X  Automation 

0.219 

0.291 

2 

14 

.490 

EE-3 


Table  EE-14.  Marking  Flight  Strips  While  Performing  Other  Tasks:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

1.000 

00.689 

1.452 

1 

7 

.267 

Table  EE-15.  Providing  Control  Information:  MANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy’s 

F 

df  1 

df2 

p-level 

Load 

.123 

0.877 

7.134 

7.134 

11.890 

3 

5 

.010 

Automation 

.061 

0.939 

15.277 

15.277 

5.092 

6 

2 

.173 

Load  X  Automation 

.072 

0.928 

12.981 

12.981 

4.327 

6 

2 

.200 

Table  EE-16.  Providing  Essential  ATC  Information:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

3m 

VBISS SW 

Load 

0.083 

49.000 

■i 

7 

Automation 

0.766 

0.980 

2 

Load  X  Automation 

0.397 

0.379 

2 

Table  EE-17.  Providing  Additional  ATC  Information:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

7.922 

0.458 

17.312 

1 

7 

.004 

Automation 

3.143 

0.567 

5.540 

2 

14 

.017 

Load  X  Automation 

0.133 

0.550 

0.242 

2 

14 

.788 

Table  EE-1 8.  Providing  Coordination:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

12.319 

1.134 

1.861 

1 

7 

.013 

Automation 

0.820 

1.093 

0.750 

2 

14 

.490 

Load  X  Automation 

2.136 

■m 

3.120 

2 

14 

.076 

Table  EE-19.  Technical  Knowledge:  MANOVA  Results. 


Wilks  A 

Pillai's 

[j£l!9 

P 

dfi 

df  2 

Load 

moM 

m 

5 

.016 

.068 

0.932 

13.798 

13.798 

4.599 

m 

m 

■1 

Load  X  Automation 

.134 

0.866 

6.478 

6.478 

2.159 

2 

.350 

Table  EE-20.  Showing  Knowledge  of  LOAs  and  SOPS:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

on 

WE/EM 

Load 

0.750 

0.238 

3.150 

■ID 

m 

HE 

Automation 

0.312 

3.610 

2 

14 

.054 

Load  X  Automation 

0.758 

0.332 

2.285 

2 

14 

.138 

Table  EE-21.  Showing  Knowledge  of  Aircraft  Capabilities  and  Limitations:  ANOVA 

Results. 


MS  Effect 

Ms  Error 

F 

iiai 

1 

Load 

■KuDl 

■ESS] 

m 

m 

HESS 

2 

m 

Load  X  Automation 

0.504 

0.157 

3.215 

2 

m 

.071 

EE-4 


Table  EE-22.  Showing  Effective  Use  of  Equipment:  ANOVA  Results. 


MS  Effect 

Ms  Error 

F 

df  1 

df  2 

p-level 

Load 

area 

m 

7 

Automation 

wi-mm 

2 

14 

Load  X  Automation 

0.256 

0.659 

0.389 

2 

14 

Table  EE-23.  Communicating:  M ANOVA  Results. 


Wilks  A 

Pillai's 

Hotelling 

Roy’s 

F 

df  1 

df  2 

p-level 

Load 

.821 

51BEM 

0.219 

0.364 

3 

5 

.782 

Automation 

.211 

3.741 

3.741 

1.247 

6 

2 

.509 

Load  X  Automation 

.254 

0.746 

2.934 

2.934 

0.978 

6 

2 

.585 

Table  EE-24.  Using  Proper  Phraseology:  ANOVA  Results. 


df  1 

ii»i 

p-level 

Load 

51 

7 

.345 

0.324 

BH 

2 

14 

.235 

maim 

2 

14 

.035 

Table  EE-25.  Communicating  Clearly  and  Efficiently:  ANOVA  Results. 


mm 

Load 

1 

m 

EE9I 

Automation 

4.243 

2 

Load  X  Automation  ; 

mmm 

5.293 

2 

Table  EE-26.  Listening  for  Pilot  Readbacks  and  Requests:  ANOVA  Results. 


F 

Iff! 

I2£l 

Load 

g£2Mi 

0.302 

1.393 

1 

7 

Automation 

mam 

anrea 

2 

m 

Load  X  Automation 

2 

m 

EE-5 


APPENDIX  FF 


Exit  Questionnaire 


Table  FF-1.  Experiment  Ratings. 


Mean 

SD 

Genera  Center  Airspace  hands-on  training 

7.63 

1.78 

Decision  Support  Tool  hands-on  training 

7.06 

2.11 

FF-1 


